Its about a 3 1/2 hr drive if you're not going to make any pit stops. Although the speed limit is mostly 65 out in the open, there are many sections with no passing, every town you go through it drops down to 40 or 30 MPH; several of those small towns run speed traps frequently, and of course you'll be going slow up & down Wolf Creek Pass.
Ditto on the time, however, you most certainly will want to stop in Durango and Pagosa Springs to take in the towns. We did a roof-top hot tub in Pagosa and then dry-camped in some little city park right across the street.