(Recasts with details from league's charges, adds background)
NEW YORK, May 20 (Reuters) - The National Basketball Association (NBA) stepped up its drive to oust Donald Sterling as owner of the Los Angeles Clippers, formally charging on Monday that his racist remarks damaged the league and paving the way for fellow team owners to decide his fate next month.
The league said Sterling has until May 27 to respond to its charge and may present his side at a special hearing before the NBA Board of Governors set for June 3, after which the league's 29 other owners who make up the board could vote to remove him.
Sterling, 80, who has owned the Clippers for 33 years, came under fire more than three weeks ago when TMZ.com posted an audio recording of him berating a female friend for publicly associating with black people, including former NBA great Earvin "Magic" Johnson.
The recording sparked an uproar from fans, NBA players and commercial sponsors. The furor led NBA Commissioner Adam Silver to ban Sterling for life from the NBA and to call on the other owners to force Sterling to relinquish team ownership.
Such an unprecedented move requires a three-fourths vote of the board.
Sterling has refused to pay a $2.5 million fine levied against him by the league and threatened through his attorney to take legal action against any forced sale of the Clippers.
Breaking his own silence about the controversy last week, Sterling apologized in a CNN interview for his "terrible mistake", while insisting he was "baited" into making provocative remarks and accusing the media of inflaming his predicament.
But he stoked further outrage during the interview by saying Magic Johnson was a poor role model for children and suggesting that Johnson and other blacks do little to support their own communities, a comment the NBA cited in its notice on Monday.
In a summary of its charge against Sterling, the league said his conduct undermined the NBA's efforts to promote diversity and inclusion; damaged its relationship with fans; harmed other NBA owners, players and Clippers personnel; and impaired the league's relationship with marketing and merchandising partners, as well as with government and community leaders.
The NBA also said its inquiry into the scandal revealed that "evidence was destroyed" and that "false and misleading evidence" was provided to league investigators.
Those acts, the NBA said, provide grounds for Sterling's termination under the NBA constitution and bylaws.
Sterling's wife, Shelly, who co-owns the team through a family trust, has said she would fight any attempt to force her to give up her interest in the club. Her lawyers released a statement on Monday saying they saw no reason why she should be forced to sell.
"She is the innocent estranged spouse," the statement said. "We also continue to hope that we can resolve this dispute with the NBA for the good of all constituencies."
The Clippers were eliminated from the second round of the NBA playoffs last Thursday by the Oklahoma City Thunder. (Reporting by Larry Fine in New York; Additional reporting and writing by Steve Gorman in Los Angeles; Editing by Gene Cherry and Matt Driskill)