The only way from DUB to Rotterdam is by flight. You should expect to be able to find tickets starting around $336, if you search flights from DUB to RTM. The best time to book shorter flights is 2-3 weeks before your trip.
On your trip from DUB to Rotterdam, it is best to book as soon as possible. It won't necessarily help reduce your price by a large amount, but will ensure that you have a spot on the bus or train. Shuttle service may be available to Rotterdam. If you're traveling with a large group, you should probably take a shuttle as it will reduce the price per person. Alternatively, there are shared shuttle services available. With this option, you'll want to book as soon as possible to allow the shuttle service enough time to coordinate your pickup and dropoff.