Rory was hysterical when she ran to Richard and it terrified him. There was fear and panic on Richards face, he had never seen Rory that way. He did not know what to do to help her, so he folded and gave in to Rory. Between her grandparents and Lorelei, Richard was the weakest link. Rory knew this and easily manipulated her way into the pool house.