Kody Brown is setting the record straight on what he believes his children resent him for.

During the latest episode of Sister Wives that aired on Sunday, Oct. 6, Kody and Robyn Brown got into a heated argument before cameras arrived that then led to Kody getting into his car and driving off by himself.

As for how the exchange began, Robyn explained, “I was putting pressure on Cody to work on his relationships with his kids and it triggered him and it upset him.”

In a confessional, Kody shares his point of view on the situation. “What has happened is in the divorce — in the divorces — is the ugly finger of blame has come out. ‘Dad, you were never at my house,’ ” he said. “I’m sorry, I had four wives. I was at your house, though, because the other wife was complaining that I was at your house.”

After his drive, the couple met outside to continue their conversation in a calmer tone.

“I just don’t know why you’re not reaching out to them more,” Robyn stressed to Kody of his children, adding that “they’re hurting.” Kody then recalled a recent family wedding, where some of his kids avoided speaking to him, insinuating that they’re the ones who don’t want a relationship with him.

“They’re purposefully leaving me out of their lives to punish me for a crime I did not commit,” he told Robyn. “I am only guilty of not falling madly in love with their mothers.”

“It’s an unrealistic expectation for grandparents to be in their grandchildren’s lives all the time,” he told the cameras during a confessional. “Especially if you move your children to an entirely different coast.”

The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!

The focus then shifted to Janelle’s daughter, Madison, as Kody claimed, “Every time I talk to her, she spreads gossip about me to the rest of the family.” He added, “Our family was killed by gossip.”

Janelle chimed in during a confessional, disproving Kody’s statement. “Cody has this narrative about Madison where she’s such the gossip, but he is as big of a gossip,” she said.

Back in the scene, Kody told Robyn, “I will work, I will reach out to, I will love those children who will allow me to do it. And in time, maybe the rest of them will come back around.”