"I found it quite difficult at times because shooting the show is a constant reminder of Lance," the actor tells EW.