content: add the notes back

5 files changed, 1447 insertions, 0 deletions

diff --git a/content/notes/containerd-to-firecracker.md b/content/notes/containerd-to-firecracker.md
new file mode 100644
index 0000000..b64586b
--- /dev/null
+++ b/content/notes/containerd-to-firecracker.md
@@ -0,0 +1,682 @@
+---
+title: containerd to firecracker
+date: 2021-05-15
+tags:
+  - linux
+  - firecracker
+  - containerd
+  - go
+---
+
+fly.io had an [interesting
+article](https://fly.io/blog/docker-without-docker/) about how they use
+docker images to create VMs for `firecracker`.
+
+They describe the process as follow:
+
+1.  Pull a container from a registry
+2.  Create a loop device to store the container's filesystem on
+3.  Unpack the container into the mounted loop device
+4.  Create a second block device and inject init, kernel, configuration
+    and other stuff
+5.  Attach persistent volumes (if any)
+6.  Create a TAP device and configure it
+7.  Hand it off to Firecracker and boot that thing
+
+That's pretty detailed, and I'm curious how difficult it is to implement
+this. I've been meaning to look into Firecracker for a while and into
+containers'd API, so this is a perfect opportunity to get started. The
+code is available [here](https://github.com/fcuny/containerd-to-vm).
+
+# #1 Pull a container from a registry with `containerd`
+
+`containerd` has a pretty [detailed
+documentation](https://pkg.go.dev/github.com/containerd/containerd).
+From the main page we can see the following example to create a client.
+
+``` go
+import (
+  "github.com/containerd/containerd"
+  "github.com/containerd/containerd/cio"
+)
+
+
+func main() {
+    client, err := containerd.New("/run/containerd/containerd.sock")
+    defer client.Close()
+}
+```
+
+And pulling an image is also pretty straightforward:
+
+``` go
+image, err := client.Pull(context, "docker.io/library/redis:latest")
+```
+
+The `Pull` method returns an
+[`Image`](https://pkg.go.dev/github.com/containerd/containerd@v1.4.4/images#Image)
+and there's a few methods associated with it.
+
+As `containerd` has namespaces, it's possible to specify the namespace
+we want to use when working with the API:
+
+``` go
+ctx := namespaces.WithNamespace(context.Background(), "c2vm")
+image, err := client.Pull(ctx, "docker.io/library/redis:latest")
+```
+
+The image will now be stored in the `c2vm` namespace. We can verify this
+with:
+
+``` bash
+; sudo ctr -n c2vm images ls -q
+docker.io/library/redis:latest
+```
+
+# #2 Create a loop device to store the container's filesystem on
+
+This is going to be pretty straightforward. To create a loop device we
+need to:
+
+1.  pre-allocate space to a file
+2.  convert that file to some format
+3.  mount it to some destination
+
+There's two commons ways to pre-allocate space to a file: `dd` and
+`fallocate` (there's likely way more ways to do this). I'll go with
+`fallocate` for this example.
+
+First, to be safe, we create a temporary file, and use `renameio` to
+handle the renaming (I recommend reading the doc of the module).
+
+``` go
+f, err := renameio.TempFile("", rawFile)
+if err != nil {
+    return err
+}
+defer f.Cleanup()
+```
+
+Now to do the pre-allocation (we're making an assumption here that 2GB
+is enough, we can likely check what's the size of the container before
+doing this):
+
+``` go
+command := exec.Command("fallocate", "-l", "2G", f.Name())
+if err := command.Run(); err != nil {
+    return fmt.Errorf("fallocate error: %s", err)
+}
+```
+
+We can now convert that file to ext4:
+
+``` go
+command = exec.Command("mkfs.ext4", "-F", f.Name())
+if err := command.Run(); err != nil {
+    return fmt.Errorf("mkfs.ext4 error: %s", err)
+}
+```
+
+Now we can rename safely the temporary file to the proper file we want:
+
+``` go
+f.CloseAtomicallyReplace()
+```
+
+And to mount that file
+
+``` go
+command = exec.Command("mount", "-o", "loop", rawFile, mntDir)
+if err := command.Run(); err != nil {
+    return fmt.Errorf("mount error: %s", err)
+}
+```
+
+# #3 Unpack the container into the mounted loop device
+
+Extracting the container using `containerd` is pretty simple. Here's the
+function that I use:
+
+``` go
+func extract(ctx context.Context, client *containerd.Client, image containerd.Image, mntDir string) error {
+    manifest, err := images.Manifest(ctx, client.ContentStore(), image.Target(), platform)
+    if err != nil {
+        log.Fatalf("failed to get the manifest: %v\n", err)
+    }
+
+    for _, desc := range manifest.Layers {
+        log.Printf("extracting layer %s\n", desc.Digest.String())
+        layer, err := client.ContentStore().ReaderAt(ctx, desc)
+        if err != nil {
+            return err
+        }
+        if err := archive.Untar(content.NewReader(layer), mntDir, &archive.TarOptions{NoLchown: true}); err != nil {
+            return err
+        }
+    }
+
+    return nil
+}
+```
+
+Calling `images.Manifest` returns the
+[manifest](https://github.com/opencontainers/image-spec/blob/master/manifest.md)
+from the image. What we care here are the list of layers. Here I'm
+making a number of assumptions regarding their type (we should be
+checking the media type first). We read the layers and extract them to
+the mounted path.
+
+# #4 Create a second block device and inject other stuff
+
+Here I'm going to deviate a bit. I will not create a second loop device,
+and I will not inject a kernel. In their article, they provided a link
+to a snapshot of their `init` process
+(<https://github.com/superfly/init-snapshot>). In order to keep this
+simple, our init is going to be a shell script composed of the content
+of the entry point of the container. We're also going to add a few extra
+files to container (`/etc/hosts` and `/etc/resolv.conf`).
+
+Finally, since we've pre-allocated 2GB for that container, and we likely
+don't need that much, we're also going to resize the image.
+
+## Add init
+
+Let's refer to the [specification for the
+config](https://github.com/opencontainers/image-spec/blob/master/config.md).
+The elements that are of interest to me are:
+
+-   `Env`, which is array of strings. They contain the environment
+    variables that likely we need to run the program
+-   `Cmd`, which is also an array of strings. If there's no entry point
+    provided, this is what is used.
+
+At this point, for this experiment, I'm going to ignore exposed ports,
+working directory, and the user.
+
+First we need to read the config from the container. This is easily
+done:
+
+``` go
+config, err := images.Config(ctx, client.ContentStore(), image.Target(), platform)
+if err != nil {
+    return err
+}
+```
+
+This needs to be read and decoded:
+
+``` go
+configBlob, err := content.ReadBlob(ctx, client.ContentStore(), config)
+var imageSpec ocispec.Image
+json.Unmarshal(configBlob, &imageSpec)
+```
+
+`init` is the first process started by Linux during boot. On a regular
+Linux desktop you likely have a symbolic link from `/usr/bin/init` to
+`/usr/lib/systemd/systemd`, since most distributions have switched to
+`systemd`. For my use case however, I want to run a single process, and
+I want it to be the one from the container. For this we can create a
+simple shell script inside the container (the location does not matter
+for now) with the environment variables and the command.
+
+Naively, this can be done like this:
+
+``` go
+initPath := filepath.Join(mntDir, "init.sh")
+f, err := renameio.TempFile("", initPath)
+if err != nil {
+    return err
+}
+defer f.Cleanup()
+
+writer := bufio.NewWriter(f)
+fmt.Fprintf(writer, "#!/bin/sh\n")
+for _, env := range initEnvs {
+    fmt.Fprintf(writer, "export %s\n", env)
+}
+fmt.Fprintf(writer, "%s\n", initCmd)
+writer.Flush()
+
+f.CloseAtomicallyReplace()
+
+mode := int(0755)
+os.Chmod(initPath, os.FileMode(mode))
+```
+
+We're once again creating a temporary file with `renamio`, and we're
+writing our shell scripts, one line at a time. We only need to make sure
+this executable.
+
+## extra files
+
+Once we have our init file, I also want to add a few extra files:
+`/etc/hosts` and `/etc/resolv.conf`. This files are not always present,
+since they can be injected by other systems. I also want to make sure
+that DNS resolutions are done using my own DNS server.
+
+## resize the image
+
+We've pre-allocated 2GB for the image, and it's likely we don't need as
+much space. We can do this by running `e2fsck` and `resize2fs` once
+we're done manipulating the image.
+
+Within a function, we can do the following:
+
+``` go
+command := exec.Command("/usr/bin/e2fsck", "-p", "-f", rawFile)
+if err := command.Run(); err != nil {
+    return fmt.Errorf("e2fsck error: %s", err)
+}
+
+command = exec.Command("resize2fs", "-M", rawFile)
+if err := command.Run(); err != nil {
+    return fmt.Errorf("resize2fs error: %s", err)
+}
+```
+
+I'm using `docker.io/library/redis:latest` for my test, and I end up
+with the following size for the image:
+
+``` bash
+-rw------- 1 root root 216M Apr 22 14:50 /tmp/fcuny.img
+```
+
+## Kernel
+
+We're going to need a kernel to run that VM. In my case I've decided to
+go with version 5.8, and build a custom kernel. If you are not familiar
+with the process, the firecracker team has [documented how to do
+this](https://github.com/firecracker-microvm/firecracker/blob/main/docs/rootfs-and-kernel-setup.md#creating-a-kernel-image).
+In my case all I had to do was:
+
+``` bash
+git clone https://github.com/torvalds/linux.git linux.git
+cd linux.git
+git checkout v5.8
+curl -o .config -s https://github.com/firecracker-microvm/firecracker/blob/main/resources/microvm-kernel-x86_64.config
+make menuconfig
+make vmlinux -j8
+```
+
+Note that they also have a pretty [good documentation for
+production](https://github.com/firecracker-microvm/firecracker/blob/main/docs/prod-host-setup.md).
+
+# #5 Attach persistent volumes (if any)
+
+I'm going to skip that step for now.
+
+# #6 Create a TAP device and configure it
+
+We're going to need a network for that VM (otherwise it might be a bit
+boring). There's a few solutions that we can take:
+
+1.  create the TAP device
+2.  delegate all that work to a
+    [CNI](https://github.com/containernetworking/cni)
+
+I've decided to use the CNI approach [documented in the Go's
+SDK](https://github.com/firecracker-microvm/firecracker-go-sdk#cni). For
+this to work we need to install the `tc-redirect-tap` CNI plugin
+(available at <https://github.com/awslabs/tc-redirect-tap>).
+
+Based on that documentation, I'll start with the following configuration
+in `etc/cni/conf.d/50-c2vm.conflist`:
+
+``` json
+{
+  "name": "c2vm",
+  "cniVersion": "0.4.0",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "c2vm-br",
+      "isDefaultGateway": true,
+      "forceAddress": false,
+      "ipMasq": true,
+      "hairpinMode": true,
+      "mtu": 1500,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "192.168.128.0/24",
+        "resolvConf": "/etc/resolv.conf"
+      }
+    },
+    {
+      "type": "firewall"
+    },
+    {
+      "type": "tc-redirect-tap"
+    }
+  ]
+}
+```
+
+# #7 Hand it off to Firecracker and boot that thing
+
+Now that we have all the components, we need to boot that VM. Since I've
+been working with Go so far, I'll also use the [Go
+SDK](https://github.com/firecracker-microvm/firecracker-go-sdk) to
+manage and start the VM.
+
+For this we need the firecracker binary, which we can [find on
+GitHub](https://github.com/firecracker-microvm/firecracker/releases).
+
+The first thing is to configure the list of devices. In our case we will
+have a single device, the boot drive that we've created in the previous
+step.
+
+``` go
+devices := make([]models.Drive, 1)
+devices[0] = models.Drive{
+    DriveID:      firecracker.String("1"),
+    PathOnHost:   &rawImage,
+    IsRootDevice: firecracker.Bool(true),
+    IsReadOnly:   firecracker.Bool(false),
+}
+```
+
+The next step is to configure the VM:
+
+``` go
+fcCfg := firecracker.Config{
+    LogLevel:        "debug",
+    SocketPath:      firecrackerSock,
+    KernelImagePath: linuxKernel,
+    KernelArgs:      "console=ttyS0 reboot=k panic=1 acpi=off pci=off i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd init=/init.sh random.trust_cpu=on",
+    Drives:          devices,
+    MachineCfg: models.MachineConfiguration{
+        VcpuCount:   firecracker.Int64(1),
+        CPUTemplate: models.CPUTemplate("C3"),
+        HtEnabled:   firecracker.Bool(true),
+        MemSizeMib:  firecracker.Int64(512),
+    },
+    NetworkInterfaces: []firecracker.NetworkInterface{
+        {
+            CNIConfiguration: &firecracker.CNIConfiguration{
+                NetworkName: "c2vm",
+                IfName:      "eth0",
+            },
+        },
+    },
+}
+```
+
+Finally we can create the command to start and run the VM:
+
+``` go
+command := firecracker.VMCommandBuilder{}.
+    WithBin(firecrackerBinary).
+    WithSocketPath(fcCfg.SocketPath).
+    WithStdin(os.Stdin).
+    WithStdout(os.Stdout).
+    WithStderr(os.Stderr).
+    Build(ctx)
+machineOpts = append(machineOpts, firecracker.WithProcessRunner(command))
+m, err := firecracker.NewMachine(vmmCtx, fcCfg, machineOpts...)
+if err != nil {
+    panic(err)
+}
+
+if err := m.Start(vmmCtx); err != nil {
+    panic(err)
+}
+defer m.StopVMM()
+
+if err := m.Wait(vmmCtx); err != nil {
+    panic(err)
+}
+```
+
+The end result:
+
+    ; sudo ./c2vm -container docker.io/library/redis:latest -firecracker-binary ./hack/firecracker/firecracker-v0.24.3-x86_64 -linux-kernel ./hack/linux/my-linux.bin -out /tmp/redis.img
+    2021/05/15 14:12:59 pulled docker.io/library/redis:latest (38690247 bytes)
+    2021/05/15 14:13:00 mounted /tmp/redis.img on /tmp/c2vm026771514
+    2021/05/15 14:13:00 extracting layer sha256:69692152171afee1fd341febc390747cfca2ff302f2881d8b394e786af605696
+    2021/05/15 14:13:00 extracting layer sha256:a4a46f2fd7e06fab84b4e78eb2d1b6d007351017f9b18dbeeef1a9e7cf194e00
+    2021/05/15 14:13:00 extracting layer sha256:bcdf6fddc3bdaab696860eb0f4846895c53a3192c9d7bf8d2275770ea8073532
+    2021/05/15 14:13:01 extracting layer sha256:b7e9b50900cc06838c44e0fc5cbebe5c0b3e7f70c02f32dd754e1aa6326ed566
+    2021/05/15 14:13:01 extracting layer sha256:5f3030c50d85a9d2f70adb610b19b63290c6227c825639b227ddc586f86d1c76
+    2021/05/15 14:13:01 extracting layer sha256:63dae8e0776cdbd63909fbd9c047c1615a01cb21b73efa87ae2feed680d3ffa1
+    2021/05/15 14:13:01 init script created
+    2021/05/15 14:13:01 umount /tmp/c2vm026771514
+    INFO[0003] Called startVMM(), setting up a VMM on firecracker.sock
+    INFO[0003] VMM logging disabled.
+    INFO[0003] VMM metrics disabled.
+    INFO[0003] refreshMachineConfiguration: [GET /machine-config][200] getMachineConfigurationOK  &{CPUTemplate:C3 HtEnabled:0xc0004e6753 MemSizeMib:0xc0004e6748 VcpuCount:0xc0004e6740}
+    INFO[0003] PutGuestBootSource: [PUT /boot-source][204] putGuestBootSourceNoContent
+    INFO[0003] Attaching drive /tmp/redis.img, slot 1, root true.
+    INFO[0003] Attached drive /tmp/redis.img: [PUT /drives/{drive_id}][204] putGuestDriveByIdNoContent
+    INFO[0003] Attaching NIC tap0 (hwaddr 9e:72:c7:04:6b:80) at index 1
+    INFO[0003] startInstance successful: [PUT /actions][204] createSyncActionNoContent
+    [    0.000000] Linux version 5.8.0 (fcuny@nas) (gcc (Debian 8.3.0-6) 8.3.0, GNU ld (GNU Binutils for Debian) 2.31.1) #1 SMP Mon Apr 12 20:07:40 PDT 2021
+    [    0.000000] Command line: i8042.dumbkbd ip=192.168.128.9::192.168.128.1:255.255.255.0:::off::: console=ttyS0 reboot=k panic=1 acpi=off pci=off i8042.noaux i8042.nomux i8042.nopnp init=/init.sh random.trust_cpu=on root=/dev/vda rw virtio_mmio.device=4K@0xd0000000:5 virtio_mmio.device=4K@0xd0001000:6
+    [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
+    [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
+    [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
+    [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
+    [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
+    [    0.000000] BIOS-provided physical RAM map:
+    [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
+    [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001fffffff] usable
+    [    0.000000] NX (Execute Disable) protection: active
+    [    0.000000] DMI not present or invalid.
+    [    0.000000] Hypervisor detected: KVM
+    [    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
+    [    0.000000] kvm-clock: cpu 0, msr 2401001, primary cpu clock
+    [    0.000000] kvm-clock: using sched offset of 11918596 cycles
+    [    0.000005] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
+    [    0.000011] tsc: Detected 1190.400 MHz processor
+    [    0.000108] last_pfn = 0x20000 max_arch_pfn = 0x400000000
+    [    0.000151] Disabled
+    [    0.000156] x86/PAT: MTRRs disabled, skipping PAT initialization too.
+    [    0.000166] CPU MTRRs all blank - virtualized system.
+    [    0.000170] x86/PAT: Configuration [0-7]: WB  WT  UC- UC  WB  WT  UC- UC
+    [    0.000201] found SMP MP-table at [mem 0x0009fc00-0x0009fc0f]
+    [    0.000257] check: Scanning 1 areas for low memory corruption
+    [    0.000364] No NUMA configuration found
+    [    0.000365] Faking a node at [mem 0x0000000000000000-0x000000001fffffff]
+    [    0.000370] NODE_DATA(0) allocated [mem 0x1ffde000-0x1fffffff]
+    [    0.000490] Zone ranges:
+    [    0.000493]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
+    [    0.000494]   DMA32    [mem 0x0000000001000000-0x000000001fffffff]
+    [    0.000495]   Normal   empty
+    [    0.000497] Movable zone start for each node
+    [    0.000500] Early memory node ranges
+    [    0.000501]   node   0: [mem 0x0000000000001000-0x000000000009efff]
+    [    0.000502]   node   0: [mem 0x0000000000100000-0x000000001fffffff]
+    [    0.000510] Zeroed struct page in unavailable ranges: 98 pages
+    [    0.000511] Initmem setup node 0 [mem 0x0000000000001000-0x000000001fffffff]
+    [    0.004990] Intel MultiProcessor Specification v1.4
+    [    0.004995] MPTABLE: OEM ID: FC
+    [    0.004995] MPTABLE: Product ID: 000000000000
+    [    0.004996] MPTABLE: APIC at: 0xFEE00000
+    [    0.005007] Processor #0 (Bootup-CPU)
+    [    0.005039] IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23
+    [    0.005041] Processors: 1
+    [    0.005042] TSC deadline timer available
+    [    0.005044] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
+    [    0.005060] KVM setup pv remote TLB flush
+    [    0.005072] KVM setup pv sched yield
+    [    0.005078] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
+    [    0.005079] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x000fffff]
+    [    0.005081] [mem 0x20000000-0xffffffff] available for PCI devices
+    [    0.005082] Booting paravirtualized kernel on KVM
+    [    0.005084] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
+    [    0.005087] setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:1 nr_node_ids:1
+    [    0.006381] percpu: Embedded 44 pages/cpu s143360 r8192 d28672 u2097152
+    [    0.006404] KVM setup async PF for cpu 0
+    [    0.006410] kvm-stealtime: cpu 0, msr 1f422080
+    [    0.006420] Built 1 zonelists, mobility grouping on.  Total pages: 128905
+    [    0.006420] Policy zone: DMA32
+    [    0.006422] Kernel command line: i8042.dumbkbd ip=192.168.128.9::192.168.128.1:255.255.255.0:::off::: console=ttyS0 reboot=k panic=1 acpi=off pci=off i8042.noaux i8042.nomux i8042.nopnp init=/init.sh random.trust_cpu=on root=/dev/vda rw virtio_mmio.device=4K@0xd0000000:5 virtio_mmio.device=4K@0xd0001000:6
+    [    0.006858] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
+    [    0.007003] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
+    [    0.007047] mem auto-init: stack:off, heap alloc:off, heap free:off
+    [    0.007947] Memory: 491940K/523896K available (10243K kernel code, 629K rwdata, 1860K rodata, 1408K init, 6048K bss, 31956K reserved, 0K cma-reserved)
+    [    0.007980] random: get_random_u64 called from __kmem_cache_create+0x3d/0x540 with crng_init=0
+    [    0.008053] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
+    [    0.008146] rcu: Hierarchical RCU implementation.
+    [    0.008147] rcu:     RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=1.
+    [    0.008151] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
+    [    0.008152] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
+    [    0.008170] NR_IRQS: 4352, nr_irqs: 48, preallocated irqs: 16
+    [    0.008373] random: crng done (trusting CPU's manufacturer)
+    [    0.008430] Console: colour dummy device 80x25
+    [    0.052276] printk: console [ttyS0] enabled
+    [    0.052685] APIC: Switch to symmetric I/O mode setup
+    [    0.053288] x2apic enabled
+    [    0.053705] Switched APIC routing to physical x2apic.
+    [    0.054213] KVM setup pv IPIs
+    [    0.055559] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x1128af0325d, max_idle_ns: 440795261011 ns
+    [    0.056516] Calibrating delay loop (skipped) preset value.. 2380.80 BogoMIPS (lpj=4761600)
+    [    0.057259] pid_max: default: 32768 minimum: 301
+    [    0.057726] LSM: Security Framework initializing
+    [    0.058176] SELinux:  Initializing.
+    [    0.058556] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
+    [    0.059221] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
+    [    0.060382] x86/cpu: User Mode Instruction Prevention (UMIP) activated
+    [    0.060510] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
+    [    0.060510] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
+    [    0.060510] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
+    [    0.060510] Spectre V2 : Mitigation: Enhanced IBRS
+    [    0.060510] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
+    [    0.060510] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
+    [    0.060510] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
+    [    0.060510] Freeing SMP alternatives memory: 32K
+    [    0.060510] smpboot: CPU0: Intel(R) Xeon(R) Processor @ 1.20GHz (family: 0x6, model: 0x3e, stepping: 0x4)
+    [    0.060510] Performance Events: unsupported p6 CPU model 62 no PMU driver, software events only.
+    [    0.060510] rcu: Hierarchical SRCU implementation.
+    [    0.060510] smp: Bringing up secondary CPUs ...
+    [    0.060510] smp: Brought up 1 node, 1 CPU
+    [    0.060510] smpboot: Max logical packages: 1
+    [    0.060523] smpboot: Total of 1 processors activated (2380.80 BogoMIPS)
+    [    0.061338] devtmpfs: initialized
+    [    0.061710] x86/mm: Memory block size: 128MB
+    [    0.062341] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
+    [    0.063245] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
+    [    0.063946] thermal_sys: Registered thermal governor 'fair_share'
+    [    0.063946] thermal_sys: Registered thermal governor 'step_wise'
+    [    0.064522] thermal_sys: Registered thermal governor 'user_space'
+    [    0.065313] NET: Registered protocol family 16
+    [    0.066398] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
+    [    0.067057] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
+    [    0.067778] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
+    [    0.068506] audit: initializing netlink subsys (disabled)
+    [    0.068708] cpuidle: using governor ladder
+    [    0.069097] cpuidle: using governor menu
+    [    0.070636] audit: type=2000 audit(1621113181.800:1): state=initialized audit_enabled=0 res=1
+    [    0.076346] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
+    [    0.077007] ACPI: Interpreter disabled.
+    [    0.077445] SCSI subsystem initialized
+    [    0.077812] pps_core: LinuxPPS API ver. 1 registered
+    [    0.078277] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
+    [    0.079206] PTP clock support registered
+    [    0.079741] NetLabel: Initializing
+    [    0.080111] NetLabel:  domain hash size = 128
+    [    0.080529] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
+    [    0.081113] NetLabel:  unlabeled traffic allowed by default
+    [    0.082072] clocksource: Switched to clocksource kvm-clock
+    [    0.082715] VFS: Disk quotas dquot_6.6.0
+    [    0.083123] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
+    [    0.083855] pnp: PnP ACPI: disabled
+    [    0.084510] NET: Registered protocol family 2
+    [    0.084718] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
+    [    0.085602] TCP established hash table entries: 4096 (order: 3, 32768 bytes, linear)
+    [    0.086365] TCP bind hash table entries: 4096 (order: 4, 65536 bytes, linear)
+    [    0.087025] TCP: Hash tables configured (established 4096 bind 4096)
+    [    0.087749] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
+    [    0.088481] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
+    [    0.089261] NET: Registered protocol family 1
+    [    0.090395] virtio-mmio: Registering device virtio-mmio.0 at 0xd0000000-0xd0000fff, IRQ 5.
+    [    0.091388] virtio-mmio: Registering device virtio-mmio.1 at 0xd0001000-0xd0001fff, IRQ 6.
+    [    0.092222] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1128af0325d, max_idle_ns: 440795261011 ns
+    [    0.093322] clocksource: Switched to clocksource tsc
+    [    0.093824] platform rtc_cmos: registered platform RTC device (no PNP device found)
+    [    0.094618] check: Scanning for low memory corruption every 60 seconds
+    [    0.095394] Initialise system trusted keyrings
+    [    0.095836] Key type blacklist registered
+    [    0.096427] workingset: timestamp_bits=36 max_order=17 bucket_order=0
+    [    0.097849] squashfs: version 4.0 (2009/01/31) Phillip Lougher
+    [    0.107488] Key type asymmetric registered
+    [    0.107905] Asymmetric key parser 'x509' registered
+    [    0.108409] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
+    [    0.109435] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
+    [    0.110116] serial8250: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
+    [    0.111877] loop: module loaded
+    [    0.112426] virtio_blk virtio0: [vda] 441152 512-byte logical blocks (226 MB/215 MiB)
+    [    0.113229] vda: detected capacity change from 0 to 225869824
+    [    0.114143] Loading iSCSI transport class v2.0-870.
+    [    0.114753] iscsi: registered transport (tcp)
+    [    0.115162] tun: Universal TUN/TAP device driver, 1.6
+    [    0.115955] i8042: PNP detection disabled
+    [    0.116498] serio: i8042 KBD port at 0x60,0x64 irq 1
+    [    0.117089] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
+    [    0.117932] intel_pstate: CPU model not supported
+    [    0.118448] hid: raw HID events driver (C) Jiri Kosina
+    [    0.119090] Initializing XFRM netlink socket
+    [    0.119555] NET: Registered protocol family 10
+    [    0.120285] Segment Routing with IPv6
+    [    0.120812] NET: Registered protocol family 17
+    [    0.121350] Bridge firewalling registered
+    [    0.122026] NET: Registered protocol family 40
+    [    0.122515] IPI shorthand broadcast: enabled
+    [    0.122961] sched_clock: Marking stable (72512224, 48198862)->(137683636, -16972550)
+    [    0.123796] registered taskstats version 1
+    [    0.124203] Loading compiled-in X.509 certificates
+    [    0.125355] Loaded X.509 cert 'Build time autogenerated kernel key: 6203e6adc37b712d3b220a26b38f3d31311d5966'
+    [    0.126355] Key type ._fscrypt registered
+    [    0.126736] Key type .fscrypt registered
+    [    0.127109] Key type fscrypt-provisioning registered
+    [    0.127657] Key type encrypted registered
+    [    0.144629] IP-Config: Complete:
+    [    0.144968]      device=eth0, hwaddr=9e:72:c7:04:6b:80, ipaddr=192.168.128.9, mask=255.255.255.0, gw=192.168.128.1
+    [    0.146044]      host=192.168.128.9, domain=, nis-domain=(none)
+    [    0.146604]      bootserver=255.255.255.255, rootserver=255.255.255.255, rootpath=
+    [    0.148347] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
+    [    0.149098] VFS: Mounted root (ext4 filesystem) on device 254:0.
+    [    0.149761] devtmpfs: mounted
+    [    0.150340] Freeing unused decrypted memory: 2040K
+    [    0.151148] Freeing unused kernel image (initmem) memory: 1408K
+    [    0.156621] Write protecting the kernel read-only data: 14336k
+    [    0.158657] Freeing unused kernel image (text/rodata gap) memory: 2044K
+    [    0.159490] Freeing unused kernel image (rodata/data gap) memory: 188K
+    [    0.160150] Run /init.sh as init process
+    462:C 15 May 2021 21:13:01.903 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
+    462:C 15 May 2021 21:13:01.904 # Redis version=6.2.3, bits=64, commit=00000000, modified=0, pid=462, just started
+    462:C 15 May 2021 21:13:01.905 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
+    462:M 15 May 2021 21:13:01.907 * Increased maximum number of open files to 10032 (it was originally set to 1024).
+    462:M 15 May 2021 21:13:01.909 * monotonic clock: POSIX clock_gettime
+                    _._
+               _.-``__ ''-._
+          _.-``    `.  `_.  ''-._           Redis 6.2.3 (00000000/0) 64 bit
+      .-`` .-```.  ```\/    _.,_ ''-._
+     (    '      ,       .-`  | `,    )     Running in standalone mode
+     |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
+     |    `-._   `._    /     _.-'    |     PID: 462
+      `-._    `-._  `-./  _.-'    _.-'
+     |`-._`-._    `-.__.-'    _.-'_.-'|
+     |    `-._`-._        _.-'_.-'    |           https://redis.io
+      `-._    `-._`-.__.-'_.-'    _.-'
+     |`-._`-._    `-.__.-'    _.-'_.-'|
+     |    `-._`-._        _.-'_.-'    |
+      `-._    `-._`-.__.-'_.-'    _.-'
+          `-._    `-.__.-'    _.-'
+              `-._        _.-'
+                  `-.__.-'
+
+    462:M 15 May 2021 21:13:01.922 # Server initialized
+    462:M 15 May 2021 21:13:01.923 * Ready to accept connections
+
+We can do a quick test with the following:
+
+``` bash
+; sudo docker run -it --rm redis redis-cli -h 192.168.128.9
+192.168.128.9:6379> get foo
+(nil)
+192.168.128.9:6379> set foo 1
+OK
+192.168.128.9:6379> get foo
+"1"
+192.168.128.9:6379>
+```
diff --git a/content/notes/making-sense-intel-amd-cpus.md b/content/notes/making-sense-intel-amd-cpus.md
new file mode 100644
index 0000000..22633af
--- /dev/null
+++ b/content/notes/making-sense-intel-amd-cpus.md
@@ -0,0 +1,191 @@
+---
+title: Making sense of Intel and AMD CPUs naming
+date: 2021-12-29
+tags:
+  - amd
+  - intel
+  - cpu
+---
+
+# Intel
+
+## Core
+
+The line up for the core family is i3, i5, i7 and i9. As of December
+2021, the current generation is Alder Lake (12th generation).
+
+The brand modifiers are:
+
+-   **i3**: laptops/low-end desktop
+-   **i5**: mainstream users
+-   **i7**: high-end users
+-   **i9**: enthusiast users
+
+How to read a SKU ? Let's use the
+[i7-12700K](https://ark.intel.com/content/www/us/en/ark/products/134594/intel-core-i712700k-processor-25m-cache-up-to-5-00-ghz.html)
+processor:
+
+-   **i7**: high end users
+-   **12**: 12th generation
+-   **700**: SKU digits, usually assigned in the order the processors
+    are developed
+-   **K**: unlocked
+
+List of suffixes:
+
+| suffix | meaning                                |
+|--------|----------------------------------------|
+| G..    | integrated graphics                    |
+| E      | embedded                               |
+| F      | require discrete graphic card          |
+| H      | high performance for mobile            |
+| HK     | high performance for mobile / unlocked |
+| K      | unlocked                               |
+| S      | special edition                        |
+| T      | power optimized lifestyle              |
+| U      | mobile power efficient                 |
+| Y      | mobile low power                       |
+| X/XE   | unlocked, high end                     |
+
+> **Unlocked,** what does that means ? A processor with the **K** suffix
+> is made with the an unlocked clock multiplier. When used with some
+> specific chipset, it's possible to overclock the processor.
+
+### Sockets/Chipsets
+
+For the Alder Lake generation, the supported socket is the
+[LGA<sub>1700</sub>](https://en.wikipedia.org/wiki/LGA_1700).
+
+For now only supported chipset for Alder Lake are:
+
+| feature                     | [z690](https://ark.intel.com/content/www/us/en/ark/products/218833/intel-z690-chipset.html) | [h670](https://www.intel.com/content/www/us/en/products/sku/218831/intel-h670-chipset/specifications.html) | [b660](https://ark.intel.com/content/www/us/en/ark/products/218832/intel-b660-chipset.html) | [h610](https://www.intel.com/content/www/us/en/products/sku/218829/intel-h610-chipset/specifications.html) |
+|-----------------------------|---------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------|
+| P and E cores over clocking | yes                                                                                         | no                                                                                                         | no                                                                                          | no                                                                                                         |
+| memory over clocking        | yes                                                                                         | yes                                                                                                        | yes                                                                                         | no                                                                                                         |
+| DMI 4 lanes                 | 8                                                                                           | 8                                                                                                          | 4                                                                                           | 4                                                                                                          |
+| chipset PCIe 4.0 lanes      | up to 12                                                                                    | up to 12                                                                                                   | up to 6                                                                                     | none                                                                                                       |
+| chipset PCIe 3.0 lanes      | up to 16                                                                                    | up to 12                                                                                                   | up to 8                                                                                     | 8                                                                                                          |
+| SATA 3.0 ports              | up to 8                                                                                     | up to 8                                                                                                    | 4                                                                                           | 4                                                                                                          |
+
+### Alder Lake (12th generation)
+
+| model      | p-cores | e-cores | GHz (base) | GHz (boosted) | TDP  |
+|------------|---------|---------|------------|---------------|------|
+| i9-12900K  | 8 (16)  | 8       | 3.2/2.4    | 5.1/3.9       | 241W |
+| i9-12900KF | 8 (16)  | 8       | 3.2/2.4    | 5.1/3.9       | 241W |
+| i7-12700K  | 8 (16)  | 4       | 3.6/2.7    | 4.9/3.8       | 190W |
+| i7-12700KF | 8 (16)  | 4       | 3.6/2.7    | 4.9/3.8       | 190W |
+| i5-12600K  | 6 (12)  | 4       | 3.7/2.8    | 4.9/3.6       | 150W |
+| i5-12600KF | 6 (12)  | 4       | 3.7/2.8    | 4.9/3.6       | 150W |
+
+-   support DDR4 and DDR5 (up to DDR5-4800)
+-   support PCIe 4.0 and 5.0 (16 PCIe 5.0 and 4 PCIe 4.0)
+
+The socket used is the [LGA
+1700](https://en.wikipedia.org/wiki/LGA_1700).
+
+Alder lake is an hybrid architecture, featuring both P-cores
+(performance cores) and E-cores (efficient cores). P-cores are based on
+the [Golden Cove](https://en.wikipedia.org/wiki/Golden_Cove)
+architecture, while the E-cores are based on the
+[Gracemont](https://en.wikipedia.org/wiki/Gracemont_(microarchitecture))
+architecture.
+
+This is a [good
+article](https://www.anandtech.com/show/16881/a-deep-dive-into-intels-alder-lake-microarchitectures/2)
+to read about this model. Inside the processor there's a microcontroller
+that monitors what each thread is doing. This can be used by the OS
+scheduler to hint on which core a thread should be scheduled on (between
+performance or efficiency).
+
+As of December 2021 this is not yet properly supported by the Linux
+kernel.
+
+## Xeon
+
+Xeon is the brand of Intel processor designed for non-consumer servers
+and workstations. The most recent generations are:
+
+-   Skylake (2017)
+-   Cascade lake (2019)
+-   Cooper lake (2020)
+
+The following brand identifiers are used:
+
+-   platinium
+-   gold
+-   silver
+-   bronze
+
+# AMD
+
+## Ryzen
+
+There are multiple generation for this brand of processors. They are
+based on the [zen micro
+architecture](https://en.wikipedia.org/wiki/Zen_(microarchitecture)).
+The current (as of December 2021) generation is Ryzen 5000.
+
+The brand modifiers are:
+
+-   ryzen 3: entry level
+-   ryzen 5: mainstream
+-   ryzen 9: high end performance
+-   ryzen 9:enthusiast
+
+List of suffixes:
+
+| suffix | meaning                                    |
+|--------|--------------------------------------------|
+| X      | high performance                           |
+| G      | integrated graphics                        |
+| T      | power optimized lifecycle                  |
+| S      | low power desktop with integrated graphics |
+| H      | high performance mobile                    |
+| U      | standard mobile                            |
+| M      | low power mobile                           |
+
+## EPYC
+
+EPYC is the AMD brand of processors for the server market, based on the
+zen architecture. They use the
+[SP3](https://en.wikipedia.org/wiki/Socket_SP3) socket. The EPYC
+processor is chipset free.
+
+## Threadripper
+
+The threadripper is for high performance desktop. It uses the
+[TR4](https://en.wikipedia.org/wiki/Socket_TR4) socket. At the moment
+there's only one chipset that supports this process, the
+[X399](https://en.wikipedia.org/wiki/List_of_AMD_chipsets#TR4_chipsets).
+
+The threadripper based on zen3 architecture is not yet released, but
+it's expected to hit the market in the first half of Q1 2022.
+
+## Sockets/Chipsets
+
+The majority of these processors use the [AM4
+socket](https://en.wikipedia.org/wiki/Socket_AM4). The threadripper line
+uses different sockets.
+
+There are multiple
+[chipset](https://en.wikipedia.org/wiki/Socket_AM4#Chipsets) for the AM4
+socket. The more advanced ones are the B550 and the X570.
+
+The threadripper processors use the TR4, sTRX4 and sWRX8 sockets.
+
+## Zen 3
+
+Zen 3 was released in November 2020.
+
+| model         | cores   | GHz (base) | GHz (boosted) | PCIe lanes | TDP  |
+|---------------|---------|------------|---------------|------------|------|
+| ryzen 5 5600x | 6 (12)  | 3.7        | 4.6           | 24         | 65W  |
+| ryzen 7 5800  | 8 (16)  | 3.4        | 4.6           | 24         | 65W  |
+| ryzen 7 5800x | 8 (16)  | 3.8        | 4.7           | 24         | 105W |
+| ryzen 9 5900  | 12 (24) | 3.0        | 4.7           | 24         | 65W  |
+| ryzen 9 5900x | 12 (24) | 3.7        | 4.8           | 24         | 105W |
+| ryzen 9 5950x | 16 (32) | 3.4        | 4.9           | 24         | 105W |
+
+-   support PCIe 3.0 and PCIe 4.0 (except for the G series)
+-   only support DDR4 (up to DDR4-3200)
diff --git a/content/notes/stuff-about-pcie.md b/content/notes/stuff-about-pcie.md
new file mode 100644
index 0000000..a3644f1
--- /dev/null
+++ b/content/notes/stuff-about-pcie.md
@@ -0,0 +1,242 @@
+---
+title: Stuff about PCIe
+date: 2022-01-03
+tags:
+  - linux
+  - harwdare
+---
+
+# Speed
+
+The most common versions are 3 and 4, while 5 is starting to be
+available with newer Intel processors.
+
+| ver | encoding  | transfer rate | x1         | x2          | x4         | x8         | x16         |
+|-----|-----------|---------------|------------|-------------|------------|------------|-------------|
+| 1   | 8b/10b    | 2.5GT/s       | 250MB/s    | 500MB/s     | 1GB/s      | 2GB/s      | 4GB/s       |
+| 2   | 8b/10b    | 5.0GT/s       | 500MB/s    | 1GB/s       | 2GB/s      | 4GB/s      | 8GB/s       |
+| 3   | 128b/130b | 8.0GT/s       | 984.6 MB/s | 1.969 GB/s  | 3.94 GB/s  | 7.88 GB/s  | 15.75 GB/s  |
+| 4   | 128b/130b | 16.0GT/s      | 1969 MB/s  | 3.938 GB/s  | 7.88 GB/s  | 15.75 GB/s | 31.51 GB/s  |
+| 5   | 128b/130b | 32.0GT/s      | 3938 MB/s  | 7.877 GB/s  | 15.75 GB/s | 31.51 GB/s | 63.02 GB/s  |
+| 6   | 128b/130  | 64.0 GT/s     | 7877 MB/s  | 15.754 GB/s | 31.51 GB/s | 63.02 GB/s | 126.03 GB/s |
+
+This is a
+[useful](https://community.mellanox.com/s/article/understanding-pcie-configuration-for-maximum-performance)
+link to understand the formula:
+
+    Maximum PCIe Bandwidth = SPEED * WIDTH * (1 - ENCODING) - 1Gb/s
+
+We remove 1Gb/s for protocol overhead and error corrections. The main
+difference between the generations besides the supported speed is the
+encoding overhead of the packet. For generations 1 and 2, each packet
+sent on the PCIe has 20% PCIe headers overhead. This was improved in
+generation 3, where the overhead was reduced to 1.5% (2/130) - see
+[8b/10b encoding](https://en.wikipedia.org/wiki/8b/10b_encoding) and
+[128b/130b encoding](https://en.wikipedia.org/wiki/64b/66b_encoding).
+
+If we apply the formula, for a PCIe version 3 device we can expect
+3.7GB/s of data transfer rate:
+
+    8GT/s * 4 lanes * (1 - 2/130) - 1G = 32G * 0.985 - 1G = ~30Gb/s -> 3750MB/s
+
+# Topology
+
+The easiest way to see the PCIe topology is with `lspci`:
+
+    $ lspci -tv
+    -[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex
+               +-01.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+               +-01.1-[01]----00.0  OCZ Technology Group, Inc. RD400/400A SSD
+               +-01.3-[02-03]----00.0-[03]----00.0  ASPEED Technology, Inc. ASPEED Graphics Family
+               +-01.5-[04]--+-00.0  Intel Corporation I350 Gigabit Network Connection
+               |            +-00.1  Intel Corporation I350 Gigabit Network Connection
+               |            +-00.2  Intel Corporation I350 Gigabit Network Connection
+               |            \-00.3  Intel Corporation I350 Gigabit Network Connection
+               +-02.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+               +-03.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+               +-04.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+               +-07.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+               +-07.1-[05]--+-00.0  Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
+               |            +-00.2  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor
+               |            \-00.3  Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller
+               +-08.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+               +-08.1-[06]--+-00.0  Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function
+               |            +-00.1  Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP
+               |            +-00.2  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
+               |            \-00.3  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller
+               +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
+               +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
+               +-18.0  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
+               +-18.1  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
+               +-18.2  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
+               +-18.3  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
+               +-18.4  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
+               +-18.5  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
+               +-18.6  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
+               \-18.7  Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
+
+# View a single device
+
+    $ lspci -s 0000:01:00.0
+    01:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. RD400/400A SSD (rev 01)
+
+# Reading `lspci` output
+
+    $ sudo lspci -vvv -s 0000:01:00.0
+    01:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. RD400/400A SSD (rev 01) (prog-if 02 [NVM Express])
+        Subsystem: OCZ Technology Group, Inc. RD400/400A SSD
+        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
+        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
+        Latency: 0, Cache Line Size: 64 bytes
+        Interrupt: pin A routed to IRQ 41
+        NUMA node: 0
+        Region 0: Memory at ef800000 (64-bit, non-prefetchable) [size=16K]
+        Capabilities: [40] Power Management version 3
+            Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
+            Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
+        Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
+            Address: 0000000000000000  Data: 0000
+        Capabilities: [70] Express (v2) Endpoint, MSI 00
+            DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
+                ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
+            DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
+                RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
+                MaxPayload 128 bytes, MaxReadReq 512 bytes
+            DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
+            LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <4us
+                ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
+            LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
+                ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
+            LnkSta: Speed 8GT/s (ok), Width x4 (ok)
+                TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
+            DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
+                 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
+                 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
+                 FRS- TPHComp- ExtTPHComp-
+                 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
+            DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
+                 AtomicOpsCtl: ReqEn-
+            LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
+            LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
+                 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
+                 Compliance De-emphasis: -6dB
+            LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
+                 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
+                 Retimer- 2Retimers- CrosslinkRes: unsupported
+        Capabilities: [b0] MSI-X: Enable+ Count=8 Masked-
+            Vector table: BAR=0 offset=00002000
+            PBA: BAR=0 offset=00003000
+        Capabilities: [100 v2] Advanced Error Reporting
+            UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
+            UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
+            UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
+            CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
+            CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
+            AERCap: First Error Pointer: 14, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
+                MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
+            HeaderLog: 05000001 0000010f 02000010 0f86d1a0
+        Capabilities: [178 v1] Secondary PCI Express
+            LnkCtl3: LnkEquIntrruptEn- PerformEqu-
+            LaneErrStat: 0
+        Capabilities: [198 v1] Latency Tolerance Reporting
+            Max snoop latency: 0ns
+            Max no snoop latency: 0ns
+        Capabilities: [1a0 v1] L1 PM Substates
+            L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+
+                  PortCommonModeRestoreTime=255us PortTPowerOnTime=400us
+            L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
+                   T_CommonMode=0us LTR1.2_Threshold=0ns
+            L1SubCtl2: T_PwrOn=10us
+        Kernel driver in use: nvme
+        Kernel modules: nvme
+
+A few things to note from this output:
+
+-   **GT/s** is the number of transactions supported (here, 8 billion
+    transactions / second). This is gen3 controller (gen1 is 2.5 and
+    gen2 is 5)xs
+-   **LNKCAP** is the capabilities which were communicated, and
+    **LNKSTAT** is the current status. You want them to report the same
+    values. If they don't, you are not using the hardware as it is
+    intended (here I'm assuming the hardware is intended to work as a
+    gen3 controller). In case the device is downgraded, the output will
+    be like this: `LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)`
+-   **width** is the number of lanes that can be used by the device
+    (here, we can use 4 lanes)
+-   **MaxPayload** is the maximum size of a PCIe packet
+
+# Debugging
+
+PCI configuration registers can be used to debug various PCI bus issues.
+
+The various registers define bits that are either set (indicated with a
+'+') or unset (indicated with a '-'). These bits typically have
+attributes of 'RW1C' meaning you can read and write them and need to
+write a '1' to clear them. Because these are status bits, if you wanted
+to 'count' the occurrences of them you would need to write some software
+that detected the bits getting set, incremented counters, and cleared
+them over time.
+
+The 'Device Status Register' (DevSta) shows at a high level if there
+have been correctable errors detected (CorrErr), non-fatal errors
+detected (UncorrErr), fata errors detected (FataErr), unsupported
+requests detected (UnsuppReq), if the device requires auxillary power
+(AuxPwr), and if there are transactions pending (non posted requests
+that have not been completed).
+
+    10000:01:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] (prog-if 02 [NVM Express])
+    ...
+            Capabilities: [100 v1] Advanced Error Reporting
+                    UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
+                    UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
+                    UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
+                    CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
+                    CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
+                    AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
+
+-   The Uncorrectable Error Status (UESta) reports error status of
+    individual uncorrectable error sources (no bits are set above):
+    -   Data Link Protocol Error (DLP)
+    -   Surprise Down Error (SDES)
+    -   Poisoned TLP (TLP)
+    -   Flow Control Protocol Error (FCP)
+    -   Completion Timeout (CmpltTO)
+    -   Completer Abort (CmpltAbrt)
+    -   Unexpected Completion (UnxCmplt)
+    -   Receiver Overflow (RxOF)
+    -   Malformed TLP (MalfTLP)
+    -   ECRC Error (ECRC)
+    -   Unsupported Request Error (UnsupReq)
+    -   ACS Violation (ACSViol)
+-   The Uncorrectable Error Mask (UEMsk) controls reporting of
+    individual errors by the device to the PCIe root complex. A masked
+    error (bit set) is not recorded or reported. Above shows no errors
+    are being masked)
+-   The Uncorrectable Severity controls whether an individual error is
+    reported as a Non-fatal (clear) or Fatal error (set).
+-   The Correctable Error Status reports error status of individual
+    correctable error sources: (no bits are set above)
+    -   Receiver Error (RXErr)
+    -   Bad TLP status (BadTLP)
+    -   Bad DLLP status (BadDLLP)
+    -   Replay Timer Timeout status (Timeout)
+    -   REPLAY NUM Rollover status (Rollover)
+    -   Advisory Non-Fatal Error (NonFatalIErr)
+-   The Correctable Erro Mask (CEMsk) controls reporting of individual
+    errors by the device to the PCIe root complex. A masked error (bit
+    set) is not reported to the RC. Above shows that Advisory Non-Fatal
+    Errors are being masked - this bit is set by default to enable
+    compatibility with software that does not comprehend Role-Based
+    error reporting.
+-   The Advanced Error Capabilities and Control Register (AERCap)
+    enables various capabilities (The above indicates the device capable
+    of generating ECRC errors but they are not enabled):
+    -   First Error Pointer identifies the bit position of the first
+        error reported in the Uncorrectable Error Status register
+    -   ECRC Generation Capable (GenCap) indicates if set that the
+        function is capable of generating ECRC
+    -   ECRC Generation Enable (GenEn) indicates if ECRC generation is
+        enabled (set)
+    -   ECRC Check Capable (ChkCap) indicates if set that the function
+        is capable of checking ECRC
+    -   ECRC Check Enable (ChkEn) indicates if ECRC checking is enabled
diff --git a/content/notes/working-with-go.md b/content/notes/working-with-go.md
new file mode 100644
index 0000000..b5e690e
--- /dev/null
+++ b/content/notes/working-with-go.md
@@ -0,0 +1,286 @@
+---
+title: Working with Go
+date: 2021-08-05
+tags:
+  - emacs
+  - go
+---
+
+*This document assumes go version \>= 1.16*.
+
+# Go Modules
+
+[Go modules](https://blog.golang.org/using-go-modules) have been added
+in 2019 with Go 1.11. A number of changes were introduced with [Go
+1.16](https://blog.golang.org/go116-module-changes). This document is a
+reference for me so that I can find answers to things I keep forgetting.
+
+## Creating a new module
+
+To create a new module, run `go mod init golang.fcuny.net/m`. This will
+create two files: `go.mod` and `go.sum`.
+
+In the `go.mod` file you'll find:
+
+-   the module import path (prefixed with `module`)
+-   the list of dependencies (within `require`)
+-   the version of go to use for the module
+
+## Versioning
+
+To bump the version of a module:
+
+``` bash
+$ git tag v1.2.3
+$ git push --tags
+```
+
+Then as a user:
+
+``` bash
+$ go get -d golang.fcuny.net/m@v1.2.3
+```
+
+## Updating dependencies
+
+To update the dependencies, run `go mod tidy`
+
+## Editing a module
+
+If you need to modify a module, you can check out the module in your
+workspace (`git clone <module URL>`).
+
+Edit the `go.mod` file to add
+
+``` go
+replace <module URL> => <path of the local checkout>
+```
+
+Then modify the code of the module and the next time you compile the
+project, the cloned module will be used.
+
+This is particularly useful when trying to debug an issue with an
+external module.
+
+## Vendor-ing modules
+
+It's still possible to vendor modules by running `go mod vendor`. This
+can be useful in the case of a CI setup that does not have access to
+internet.
+
+## Proxy
+
+As of version 1.13, the variable `GOPROXY` defaults to
+`https://proxy.golang.org,direct` (see
+[here](https://github.com/golang/go/blob/c95464f0ea3f87232b1f3937d1b37da6f335f336/src/cmd/go/internal/cfg/cfg.go#L269)).
+As a result, when running something like
+`go get golang.org/x/tools/gopls@latest`, the request goes through the
+proxy.
+
+There's a number of ways to control the behavior, they are documented
+[here](https://golang.org/ref/mod#private-modules).
+
+There's a few interesting things that can be done when using the proxy.
+There's a few special URLs (better documentation
+[here](https://golang.org/ref/mod#goproxy-protocol)):
+
+| path                  | description                                                                              |
+|-----------------------|------------------------------------------------------------------------------------------|
+| $mod/@v/list          | Returns the list of known versions - there's one version per line and it's in plain text |
+| $mod/@v/$version.info | Returns metadata about a version in JSON format                                          |
+| $mod/@v/$version.mod  | Returns the `go.mod` file for that version                                               |
+
+For example, looking at the most recent versions for `gopls`:
+
+``` bash
+; curl -s -L https://proxy.golang.org/golang.org/x/tools/gopls/@v/list|sort -r|head
+v0.7.1-pre.2
+v0.7.1-pre.1
+v0.7.1
+v0.7.0-pre.3
+v0.7.0-pre.2
+v0.7.0-pre.1
+v0.7.0
+v0.6.9-pre.1
+v0.6.9
+v0.6.8-pre.1
+```
+
+Let's check the details for the most recent version
+
+``` bash
+; curl -s -L https://proxy.golang.org/golang.org/x/tools/gopls/@v/list|sort -r|head
+v0.7.1-pre.2
+v0.7.1-pre.1
+v0.7.1
+v0.7.0-pre.3
+v0.7.0-pre.2
+v0.7.0-pre.1
+v0.7.0
+v0.6.9-pre.1
+v0.6.9
+v0.6.8-pre.1
+```
+
+And let's look at the content of the `go.mod` for that version too:
+
+``` bash
+; curl -s -L https://proxy.golang.org/golang.org/x/tools/gopls/@v/v0.7.1-pre.2.mod
+module golang.org/x/tools/gopls
+
+go 1.17
+
+require (
+        github.com/BurntSushi/toml v0.3.1 // indirect
+        github.com/google/go-cmp v0.5.5
+        github.com/google/safehtml v0.0.2 // indirect
+        github.com/jba/templatecheck v0.6.0
+        github.com/sanity-io/litter v1.5.0
+        github.com/sergi/go-diff v1.1.0
+        golang.org/x/mod v0.4.2
+        golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect
+        golang.org/x/sys v0.0.0-20210510120138-977fb7262007
+        golang.org/x/text v0.3.6 // indirect
+        golang.org/x/tools v0.1.6-0.20210802203754-9b21a8868e16
+        golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
+        honnef.co/go/tools v0.2.0
+        mvdan.cc/gofumpt v0.1.1
+        mvdan.cc/xurls/v2 v2.2.0
+)
+```
+
+# Tooling
+
+## LSP
+
+`gopls` is the default implementation of the language server protocol
+maintained by the Go team. To install the latest version, run
+`go install golang.org/x/tools/gopls@latest`
+
+## `staticcheck`
+
+[`staticcheck`](https://staticcheck.io/) is a great tool to run against
+your code to find issues. To install the latest version, run
+`go install honnef.co/go/tools/cmd/staticcheck@latest`.
+
+# Emacs integration
+
+## `go-mode`
+
+[This is the mode](https://github.com/dominikh/go-mode.el) to install to
+get syntax highlighting (mostly).
+
+## Integration with LSP
+
+Emacs has a pretty good integration with LSP, and ["Eglot for better
+programming experience in
+Emacs"](https://whatacold.io/blog/2022-01-22-emacs-eglot-lsp/) is a good
+starting point.
+
+### eglot
+
+[This is the main mode to install](https://github.com/joaotavora/eglot).
+
+The configuration is straightforward, this is what I use:
+
+``` elisp
+;; for go's LSP I want to use staticcheck and placeholders for completion
+(customize-set-variable 'eglot-workspace-configuration
+                        '((:gopls .
+                                  ((staticcheck     . t)
+                                   (matcher         . "CaseSensitive")
+                                   (usePlaceholders . t)))))
+
+;; ensure we load eglot for some specific modes
+(dolist (hook '(go-mode-hook nix-mode-hook))
+  (add-hook hook 'eglot-ensure))
+```
+
+`eglot` integrates well with existing modes for Emacs, mainly xref,
+flymake, eldoc.
+
+# Profiling
+
+## pprof
+
+[pprof](https://github.com/google/pprof) is a tool to visualize
+performance data. Let's start with the following test:
+
+``` go
+package main
+
+import (
+    "strings"
+    "testing"
+)
+
+func BenchmarkStringJoin(b *testing.B) {
+    input := []string{"a", "b"}
+    for i := 0; i <= b.N; i++ {
+        r := strings.Join(input, " ")
+        if r != "a b" {
+            b.Errorf("want a b got %s", r)
+        }
+    }
+}
+```
+
+Let's run a benchmark with
+`go test . -bench=. -cpuprofile cpu_profile.out`:
+
+``` go
+goos: linux
+goarch: amd64
+pkg: golang.fcuny.net/m
+cpu: Intel(R) Core(TM) i3-1005G1 CPU @ 1.20GHz
+BenchmarkStringJoin-4           41833486                26.85 ns/op            3 B/op          1 allocs/op
+PASS
+ok      golang.fcuny.net/m      1.327s
+```
+
+And let's take a look at the profile with
+`go tool pprof cpu_profile.out`
+
+``` bash
+File: m.test
+Type: cpu
+Time: Aug 15, 2021 at 3:01pm (PDT)
+Duration: 1.31s, Total samples = 1.17s (89.61%)
+Entering interactive mode (type "help" for commands, "o" for options)
+(pprof) top
+Showing nodes accounting for 1100ms, 94.02% of 1170ms total
+Showing top 10 nodes out of 41
+      flat  flat%   sum%        cum   cum%
+     240ms 20.51% 20.51%      240ms 20.51%  runtime.memmove
+     220ms 18.80% 39.32%      320ms 27.35%  runtime.mallocgc
+     130ms 11.11% 50.43%      450ms 38.46%  runtime.makeslice
+     110ms  9.40% 59.83%     1150ms 98.29%  golang.fcuny.net/m.BenchmarkStringJoin
+     110ms  9.40% 69.23%      580ms 49.57%  strings.(*Builder).grow (inline)
+     110ms  9.40% 78.63%     1040ms 88.89%  strings.Join
+      70ms  5.98% 84.62%      300ms 25.64%  strings.(*Builder).WriteString
+      50ms  4.27% 88.89%      630ms 53.85%  strings.(*Builder).Grow (inline)
+      40ms  3.42% 92.31%       40ms  3.42%  runtime.nextFreeFast (inline)
+      20ms  1.71% 94.02%       20ms  1.71%  runtime.getMCache (inline)
+```
+
+We can get a breakdown of the data for our module:
+
+``` bash
+(pprof) list golang.fcuny.net
+Total: 1.17s
+ROUTINE ======================== golang.fcuny.net/m.BenchmarkStringJoin in /home/fcuny/workspace/gobench/app_test.go
+     110ms      1.15s (flat, cum) 98.29% of Total
+         .          .      5:   "testing"
+         .          .      6:)
+         .          .      7:
+         .          .      8:func BenchmarkStringJoin(b *testing.B) {
+         .          .      9:   b.ReportAllocs()
+      10ms       10ms     10:   input := []string{"a", "b"}
+         .          .     11:   for i := 0; i <= b.N; i++ {
+      20ms      1.06s     12:           r := strings.Join(input, " ")
+      80ms       80ms     13:           if r != "a b" {
+         .          .     14:                   b.Errorf("want a b got %s", r)
+         .          .     15:           }
+         .          .     16:   }
+         .          .     17:}
+```
diff --git a/content/notes/working-with-nix.md b/content/notes/working-with-nix.md
new file mode 100644
index 0000000..9e697d5
--- /dev/null
+++ b/content/notes/working-with-nix.md
@@ -0,0 +1,46 @@
+---
+title: working with nix
+date: 2022-05-10
+tags:
+  - linux
+  - nix
+---
+
+# the `nix develop` command
+
+The `nix develop` command is for working on a repository. If our
+repository contains a `Makefile`, it will be used by the various
+sub-commands.
+
+`nix develop` supports multiple
+[phases](https://nixos.org/manual/nixpkgs/stable/#sec-stdenv-phases) and
+they map as follow:
+
+| phase          | default to     | command                   | note |
+|----------------|----------------|---------------------------|------|
+| configurePhase | `./configure`  | `nix develop --configure` |      |
+| buildPhase     | `make`         | `nix develop --build`     |      |
+| checkPhase     | `make check`   | `nix develop --check`     |      |
+| installPhase   | `make install` | `nix develop --install`   |      |
+
+In the repository, running `nix develop --build` will build the binary
+**using the Makefile**. This is different from running `nix build`.
+
+# the `nix build` and `nix run` commands
+
+## for Go
+
+For Go, there's the `buildGoModule`. Looking at the
+[source](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/go-modules/generic/default.nix)
+we can see there's a definition of what will be done for each phases. As
+a result, we don't have to define them ourselves.
+
+If we run `nix build` in the repository, it will run the default [build
+phase](https://github.com/NixOS/nixpkgs/blob/fb7287e6d2d2684520f756639846ee07f6287caa/pkgs/development/go-modules/generic/default.nix#L171).
+
+# `buildInputs` or `nativeBuildInputs`
+
+-   `nativeBuildInputs` is intended for architecture-dependent
+    build-time-only dependencies
+-   `buildInputs` is intended for architecture-independent
+    build-time-only dependencies