From fabae5c23ad8be3840c76c237ad9a8c5f0f40616 Mon Sep 17 00:00:00 2001 From: Franck Cuny Date: Sat, 8 Jan 2022 10:07:36 -0800 Subject: blog: add content about PCIe --- content/notes/stuff-about-pcie.org | 196 +++++++++++++++++++++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 content/notes/stuff-about-pcie.org (limited to 'content/notes') diff --git a/content/notes/stuff-about-pcie.org b/content/notes/stuff-about-pcie.org new file mode 100644 index 0000000..4d1a825 --- /dev/null +++ b/content/notes/stuff-about-pcie.org @@ -0,0 +1,196 @@ +#+TITLE: Stuff about PCIe +#+DATE: <2022-01-03 Mon> +#+TAGS[]: linux hardware +#+toc: t + +* Speed +The most common versions are 3 and 4, while 5 is starting to be available with newer Intel processors. + +| ver | encoding | transfer rate | x1 | x2 | x4 | x8 | x16 | +|-----+-----------+---------------+------------+-------------+------------+------------+-------------| +| 1 | 8b/10b | 2.5GT/s | 250MB/s | 500MB/s | 1GB/s | 2GB/s | 4GB/s | +| 2 | 8b/10b | 5.0GT/s | 500MB/s | 1GB/s | 2GB/s | 4GB/s | 8GB/s | +| 3 | 128b/130b | 8.0GT/s | 984.6 MB/s | 1.969 GB/s | 3.94 GB/s | 7.88 GB/s | 15.75 GB/s | +| 4 | 128b/130b | 16.0GT/s | 1969 MB/s | 3.938 GB/s | 7.88 GB/s | 15.75 GB/s | 31.51 GB/s | +| 5 | 128b/130b | 32.0GT/s | 3938 MB/s | 7.877 GB/s | 15.75 GB/s | 31.51 GB/s | 63.02 GB/s | +| 6 | 128b/130 | 64.0 GT/s | 7877 MB/s | 15.754 GB/s | 31.51 GB/s | 63.02 GB/s | 126.03 GB/s | + +This is a [[https://community.mellanox.com/s/article/understanding-pcie-configuration-for-maximum-performance][useful]] link to understand the formula: Maximum PCIe Bandwidth = *SPEED* * *WIDTH* * (1 - ENCODING) - 1Gb/s. + +We remove 1Gb/s for protocol overhead and error corrections. The main difference between the generations besides the supported speed is the encoding overhead of the packet. For generations 1 and 2, each packet sent on the PCIe has 20% PCIe headers overhead. This was improved in generation 3, where the overhead was reduced to 1.5% (2/130) - see [[https://en.wikipedia.org/wiki/8b/10b_encoding][8b/10b encoding]] and [[https://en.wikipedia.org/wiki/64b/66b_encoding][128b/130b encoding]]. + +If we apply the formula, for a PCIe version 3 device we can expect 3.7GB/s of data transfer rate: +#+begin_src +8GT/s * 4 lanes * (1 - 2/130) - 1G = 32G * 0.985 - 1G = ~30Gb/s -> 3750MB/s +#+end_src +* Topology +The easiest way to see the PCIe topology is with =lspci=: +#+begin_src +$ lspci -tv +-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex + +-01.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge + +-01.1-[01]----00.0 OCZ Technology Group, Inc. RD400/400A SSD + +-01.3-[02-03]----00.0-[03]----00.0 ASPEED Technology, Inc. ASPEED Graphics Family + +-01.5-[04]--+-00.0 Intel Corporation I350 Gigabit Network Connection + | +-00.1 Intel Corporation I350 Gigabit Network Connection + | +-00.2 Intel Corporation I350 Gigabit Network Connection + | \-00.3 Intel Corporation I350 Gigabit Network Connection + +-02.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge + +-03.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge + +-04.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge + +-07.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge + +-07.1-[05]--+-00.0 Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function + | +-00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor + | \-00.3 Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller + +-08.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge + +-08.1-[06]--+-00.0 Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function + | +-00.1 Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP + | +-00.2 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] + | \-00.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller + +-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller + +-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge + +-18.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 + +-18.1 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 + +-18.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 + +-18.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 + +-18.4 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 + +-18.5 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 + +-18.6 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 + \-18.7 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 +#+end_src +* View a single device +#+begin_src +$ lspci -s 0000:01:00.0 +01:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. RD400/400A SSD (rev 01) +#+end_src +* Reading =lspci= output +#+begin_src +$ sudo lspci -vvv -s 0000:01:00.0 +01:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. RD400/400A SSD (rev 01) (prog-if 02 [NVM Express]) + Subsystem: OCZ Technology Group, Inc. RD400/400A SSD + Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ + Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-