Last year (from Toronto), my vacation was basically half of Wednesday, Thursday, Friday, and Monday. As it happened, we got back late on Sunday since the car broke Friday night. We stopped overnight both ways.
I'm guessing you'll need Wed/Thur and Mon as travel days. It is worth the trip.