Nvidia interconects: Mellanox, NVLink and NVSwitch
By annoucing the new ARM-based Grace CPU at GTC2022 which will be available in 2023, Nvidia can again provide fast links between CPU and GPUs as they will be connected via their upgraded NVLink technology. Nvidia also annouced in March 2019 that they had reached an agreement with  Mellanox Technologies to buy the company for USD 6.9 Billion \cite{billion}. The Israeli-American company is known for its Infiniband technology used betwenn high-end servers. This has lead Nvidia to develope their interconnected technology further.
Their fourth generation NVIDIA® NVLink® technology provides 1.5X higher bandwidth and improved scalability for multi-GPU system configurations. A single NVIDIA H100 Tensor Core GPU supports up to 18 NVLink connections for a total bandwidth of 900 gigabytes per second (GB/s)—over 7X the bandwidth of PCIe Gen5. 
The NVLink Switch System can support clusters of up to 256 connected H100s and promises to delivers 9X higher bandwidth than InfiniBand HDR on Ampere. In addition, Nvidia is taking advantage of its Mellanox purchase with NVLink now supporting in-network computing called SHARP, previously only available on Infiniband. Nvidia states it will deliver one exaFLOP of FP8 compute while delivering 57.6 terabytes/s (TB/s) of All2All bandwidth.