WASHINGTON—The Securities and Exchange Commission has tried for several years to get Elon Musk to behave more like a typical corporate leader. It hasn’t worked.

The SEC sued Mr. Musk and Tesla Inc. in 2018 over tweets the regulator deemed fraudulent, settling the case for a combined $40 million and changes to the company’s corporate governance. As part of the deal, it also fashioned a unique way to restrain Mr. Musk’s social-media habit: forcing the company to preapprove any statement by its CEO that could move markets.

The Wall Street Journal reported Tuesday that SEC officials wrote Tesla twice, in 2019 and 2020, arguing that some of Mr. Musk’s tweets should have undergone the required oversight. Tesla disagreed, telling the SEC that his messages weren’t within the scope of the agreement.

There was no precedent for the communications-oversight policy in financial enforcement, although regulators often prod companies to improve compliance in the wake of scandals or missteps. Restraining how a brash CEO communicates with the public and investors has only led to more feuds with Mr. Musk and Tesla, who seem to understand the leverage they have.

“Because Elon is so intertwined with this company, any serious enforcement action against Musk has the potential to harm the company and shareholders,” said David Rosenfeld, a former SEC official now teaching law at Northern Illinois University. “It could end up being counterproductive. Musk has been poking the bear because he knows he can get away with it.”

Regulators say Tesla never followed through with oversight, while Mr. Musk has continued to tweet in a way that the SEC saw as violating the court-ordered policy, according to correspondence obtained this week by the Journal under a Freedom of Information Act request. Tesla didn’t respond to a request for comment.

Many small shareholders revel in Mr. Musk’s willingness to stiff-arm regulators. Others are weary of the spectacle. Ross Gerber, chief executive of Gerber Kawasaki Wealth and Investment Management, a Tesla shareholder, said he would like to see the company’s board of directors step in to moderate Mr. Musk’s behavior on Twitter.

“I’m disappointed with the lack of oversight and I’m annoyed that Elon seems to feel that he’s exempt from any criticism for his behavior,” said Mr. Gerber, whose firm has about $1.85 billion in assets under management.

To get the result it wants, the SEC could double down on the policy by going back to federal court and asking a judge to hold Mr. Musk in contempt for violating it. The regulator also could try to unwind the whole deal and resume its fraud lawsuit against Mr. Musk, with the potential to have him barred from public markets. Neither option is appealing, according to securities lawyers and legal researchers.

The SEC’s previous attempt in early 2019 to get U.S. District Judge Alison Nathan to force Mr. Musk and Tesla to follow the agreement didn’t work out as planned. Judge Nathan told the two sides to “put their reasonableness pants on” and settle the matter.

In response, the SEC and Tesla agreed in April 2019 to narrow the list of items required for preapproval. While the first deal called for oversight of any written communication that could be material to shareholders, the second version specified eight or nine topics that Tesla’s lawyers had to review before Mr. Musk could tweet about them, including statements about production numbers, projections, and finances.

The outcome was “not a clear win” for the SEC, said Urska Velikonja, a law professor at Georgetown University who focuses her research on securities enforcement. The agency’s lawyers are likely hesitant to return to the same judge and risk losing, Ms. Velikonja said.

The SEC has to be careful to not go too far and stifle an executive from sharing useful information with markets, said Donald Langevoort, who also teaches securities law at Georgetown. “I don’t think anyone—the judge, the SEC, or Tesla’s board—relishes the task of muzzling Elon Musk, and it seems that each of these wants someone else to deal with it,” he said.

Given their hesitancy to litigate, regulators have few options left. Senior SEC officials decided last year to stop writing to the company, convinced it wasn’t working and made the agency look powerless, according to a person familiar with the matter.

One weapon available to the SEC, Ms. Velikonja said, is seeking to have Mr. Musk barred from serving as an officer or director of a public company. That punishment is a more common penalty in SEC enforcement cases, but it requires showing a person violated antifraud laws and is unfit to serve in a leadership role.

SEC obtained bars in 32 cases during its 2019 fiscal year, Ms. Velikonja said. Elizabeth Holmes, the founder of blood-testing startup Theranos Inc., agreed in March 2018 to be suspended from officer or director roles for a decade.

But Theranos was a private firm, while Tesla is an established public company with a broad base of public shareholders. Barring Mr. Musk from the company could hurt its future prospects and market value, risking shareholder harm, lawyers said. Absent fresh allegations of fraud by Mr. Musk, regulators probably wouldn’t try it.

“There is no way the SEC wants to touch that with a 10-foot pole,” Ms. Velikonja said.

Write to Dave Michaels at dave.michaels@wsj.com and Rebecca Elliott at rebecca.elliott@wsj.com