Order your Uber 20 minutes before you need it. If he shows up before you’re ready make him wait.
Next time you get this guy, let him know your situation. Maybe get his direct number so you can call him way ahead of time to let him know you need a ride and, wow!, make an appointment?
Use a little Ju Jitsu and turn this situation around to your benefit. Because this driver is so near to you, you actually have a private driver.
Because of bonus terms he may have been driving for 10 or 12 hours and trying to get some rest but still leave the app on to accumulate hour, and maybe it takes him a while to saddle up. Or maybe he’s lazy?
There are a lot of good and bad reasons it’s taking so long. But please don’t ding this guy for a job that pays him less than $12/hr and and taking on a ton of risk to give you a very cheap ride wherever you want to go.