

I agree that hitting swap is not ideal, but messing with swappiness sysctl is almost never the answer and very often leads to more problems than it solves.
If you want to tune for workload without needing fairly deep understanding of how Linux manages memory, use a sysctl that sets a whole system behaviour like CPU governor presets.
IMO, Linux is configured for “old” systems by default: slow HDDs, and constrained RAM pools, where disk IO really, really needs caching, and where idle background processes take a large fraction of RAM.
Not really a matter of opinion, and not true since kernel 5.1 when ssds became first class citizens.
No, sorry. We don’t know enough to say this. What were the options when configuring ext4? What record size is grafana using? Mariadb or postgres? Is there contention with another db? Is there a mounted NFS path in the same ext4 partition with weird block settings? There are lots of legitimate config reasons for op’s problem, but there are so many parts of op’s setup we don’t know, we similarly can’t possibly know what the issue is.
Swappiness works as intended in most default scheduler configurations and seeing high swap is a symptom, not an issue. It is almost always more productive to find out why the swapping is happening.
Not sure what you’re getting at with TRIM.