Sean Penn's 'Mind Doesn't Quiet Down,' Source Says as Actor Admits to Using Ambien

Sean Penn’s bizarre late-night appearance this week might have surprised some fans, but those close to him say his unpredictable behavior is actually par for the course.

“His mind doesn’t quiet down,” a source close to the actor tells PEOPLE. “So he can seem erratic to people who are not used to his energy.”

Penn has been busy promoting his first novel on Marc Maron’s WTF podcast, The Daily Show with Trevor Noah and Conan. It was a recent appearance on the Late Show with Stephen Colbert thatraised eyebrows. During the interview, the actor, 57, smoked two cigarettes on-air and admitted to Colbert that he had taken a sedative before the interview began.

“He can get in an agitated state, but he’s extremely smart and very funny,” says the source, who notes that Penn compensates for some of his bad habits by eating well and getting lots of exercise.

“He actually eats well,” the source explains. “He works out — he’s jacked! He goes to holistic doctors. But it’s a dichotomy. He drinks and smokes and doesn’t sleep.”

Penn is not concerned with the public’s reaction to him, according to another source. “Sean doesn’t care at all what people think,” a source previously told PEOPLE. “He likes controversy and debate and doesn’t mind shocking people.”

According to a third source, “Sean is a one man band, always has been. He loves the fight.”

Meanwhile, Penn has been busy promoting his new novel, Bob Honey Who Just Do Stuff, and spending time with his kids with ex-wife Robin Wright, Dylan, 26, and Hopper, 24. “He’s very interested in his kids,” says the first source. “He’d drop everything for them.”

As for his love life, Penn told Marc Maron that he hasn’t given up on finding the right person. “I’m never going to take a position that I’ve closed off to love,” he said. “I think people falling in love with each other is a great, great thing.”

He was recently spotted on a dinner date with Amber Heard, but the first source says the two are not in a relationship. “Amber and Sean are friends, but they’re not dating,” says the source. “They’ve gone out to dinner.”

He’s also “still very close with” his ex-wife Madonna, adds the source.