To protect your server, the best thing you can do is configure the website (or websites) (or all the domains on the VPS) receiving attacks with the free version of Cloudflare. This will protect you from DoS and DDoS attacks, and your server will be “invisible” from the internet.
Once this is done, you will only need to worry about SMTP, which is somewhat easier to deal with using tools such as fail2ban, among others.