I know what you're talking about, you explained it better in the server and I've had this happen to me before. Randomly your weapons like to go invisible. I'm not sure what causes it, and it's also pretty rare. I've come across it on other gmod servers and I think it's a gmod issue, not just a Devinity issue. However to fix it record a demo in console and stop it right after. ex: record 1 stop
It's not only the physgun, it will removes the viewmodel on all other weapons during rounds aswell. It's very rare tho, don't have any screenshots sadly but I can take it next time it happens. If you get this issue, just rejoining will fix it!