BetaDoggo_@lemmy.worldtoSelfhosted@lemmy.world•Anyone else wants a one box replacement to a switch attached to USB NICs?English
21·
1 year agoI’ve used the tplink ones that they’re using and they’ve been pretty solid. I can’t say how they’d fare in a 24/7 setup though since they’re not really intended for that.
Koboldcpp should allow you to run much larger models with a little bit of ram offloading. There’s a fork that supports rocm for AMD cards: https://github.com/YellowRoseCx/koboldcpp-rocm
Make sure to use quantized models for the best performace, q4k_M being the standard.