* Kingdom pursuing more aggressive foreign policy
* Riyadh uneasy about Iran nuclear deal, perceived meddling
* After executions, Tehran warns of repercussions
* Ideological divide sets up simmering mistrust
* Scope for further escalation of tension exists
By Angus McDowall
RIYADH, Jan 5 (Reuters) - The last time Saudi Arabia broke off ties with Iran, after its embassy in Tehran was stormed by protesters in 1988, it took a swing in the regional power balance in the form of Saddam Hussein's 1990 invasion of Kuwait to heal the rift.
It is hard to see how any lesser development could resolve the region's most bitter rivalry, which has underpinned wars and political tussles across the Middle East as Riyadh and Tehran backed opposing sides.
Riyadh's expulsion of Iran's envoy after another storming of its Tehran embassy, this time in response to the Saudi execution of Shi'ite Muslim cleric Nimr al-Nimr, raised the heat again, making the region's underlying conflict even harder to resolve.
At the heart of the new crisis is Saudi Arabia's growing willingness to confront Iran and its allies militarily since King Salman took power a year ago, say diplomats, choosing with his son, Deputy Crown Prince Mohammed bin Salman, to abandon years of backroom politics.
Last year, Riyadh began a war in Yemen to stop an Iran-allied militia seizing power in its southern neighbour and boosted support to Syrian rebels against Tehran's ally President Bashar al-Assad. Its execution of Nimr, while mainly driven by domestic politics, was also part of that open confrontation with Iran, according to political analysts.
The interventions followed years of Riyadh complaining about what it regarded as unchecked Iranian aggression in the region. It has pointed to Iran's support for Shi'ite militias and accused the country of smuggling arms to groups in Gulf countries - which Iran denies.
"We will not allow Iran to destabilise our region. We will not allow Iran to do harm to our citizens or those of our allies and so we will react," Saudi Foreign Minister Adel al-Jubeir told Reuters on Monday, signalling Riyadh would not back down.
The Saudi decisions in Syria and Yemen were also partly a response to Iran's nuclear deal with world powers, which lifted sanctions on Tehran, theoretically giving it more money and political room to pursue its regional activities.
The new crisis has had the effect of hardening a wider confrontation between the loose-knit coalitions of allies each can call upon in the region; some of Riyadh's allies also cut diplomatic ties with Tehran after the embassy attack, while Iran's warned of repercussions.
That chain reaction may now complicate complex political talks over the formation of a government in Lebanon, efforts to bring Syria's warring parties to talks, stalled negotiations to end Yemen's civil war and Riyadh's rapprochement with Baghdad.
Until the 1960s and 70s, Saudi Arabia and Iran were uneasy allies regarded as the "twin pillars" of Washington's strategy to curb Soviet influence in the Gulf. Sectarianism was muted.
But rich on its new oil wealth, Saudi Arabia began to propagate its rigid Salafi interpretation of Sunni Islam which regards Shi'ism as heretical, in mosques around the region. And, after its 1979 revolution, Iran adopted - and exported - the doctrine of Velayat-e Faqih, which says ultimate temporal power among Shi'ites should reside with its own supreme leader.
That growing ideological divide set up a simmering mistrust that was soon matched by a geopolitical rivalry that has driven their fractious relations for the subsequent 37 years.
After Iran's 1980-88 war with Iraq, when Saddam invaded, it developed a strategy of "forward defence", seeking to use ties with Arab Shi'ites to build militias and political parties that could stop new enemies emerging and give it deterrent capability through proxy forces.
Riyadh regarded Tehran's cultivation of Shi'ite groups with intense suspicion, fearing it would foment revolution in Saudi-allied states and destabilise the region.
It broke ties in 1988 when a diplomat died in the storming of its Tehran embassy following tensions over the death of hundreds of Iranian pilgrims in clashes with Saudi police during the haj. But when Saddam invaded Kuwait, Tehran and Riyadh set aside their hostility to make common cause against a shared enemy.
The toppling of Saddam in 2003 upturned the regional power balance, however, as Iran used its ties to the country's large Shi'ite community to gain sway in Baghdad, pitting Riyadh and Tehran more openly against each other - a pattern repeated in Yemen and Syria after the "Arab Spring" uprisings.
Meanwhile, Iraq's civil war had poured fuel on growing sectarian tensions as al Qaeda, which follows an extreme form of Salafism, sent suicide bombers against Shi'ite civilians, prompting murderous retaliation from Iran-linked militias.
Now there is some scope for further escalation, both in the various Middle East theatres where Iran and Saudi Arabia back opposing forces, and diplomatically as Riyadh taps Arab and Muslim channels to try to isolate Tehran, according to analysts.
"Since 1979 the two countries have fought numerous proxy conflicts throughout the Middle East and often exchange threats and insults. But they've stopped short of direct conflict and eventually agreed to a cold reconciliation," said Karim Sadjadpour, senior associate of the Carnegie Endowment for International Peace's Middle East programme.
But he said that Iran might seek to stoke unrest among Saudi Arabia and Bahrain's Shi'ite communities.
Renewed protests among Saudi and Bahraini Shi'ites since the execution of Nimr, along with the bombing of two Sunni mosques in Iraq, may be regarded by Riyadh as evidence of Iranian incitement.
Riyadh has itself pushed allies to cut ties with Iran and pressed Muslim bodies like the Organisation of Islamic Cooperation to condemn the storming of the embassy. Theoretically, it could also ramp up support for Syrian rebel groups.
All-out conflict is something that even hawks in Saudi Arabia and Iran would likely try to avoid. However, the new escalation between the region's main enemies shows how events can sometimes pre-empt strategic plans.
After the execution of Nimr, Iran's Revolutionary Guard elite military force said a "harsh revenge" would strike Saudi's ruling Al-Saud family in the near future.
"The Revolutionary Guard is part of the Iranian government and their threats should be taken seriously because they control militia in Lebanon, Syria, Iraq and Yemen and I would not be surprised if they use it to act against the Saudis," said Abdulaziz al-Sager, head of Jeddah-based Gulf Research Centre.
(Editing by William Maclean and Pravin Char)