Combat triangle

In this image, each fighter is attacking the one weaker than it and being attacked with its own weakness

The Combat triangle is a chart of the strengths and weaknesses of each combat class against the other classes. In general, when players are combatting opponents with skills close to their own skills:

Warriors are strong against rangers. Rangers' projectiles can't penetrate metal armour very well, but warriors' blades can cut through leather with ease.
Rangers are strong against mages. Their leather and dragonhide armours absorb the magic spells used by mages. Their arrows easily pierce the soft cloth of mage robes.
Mages are strong against warriors. The metal that makes warriors' armour conducts spells so that the effect is more potent. Also, freeze/bind spells put warriors at a disadvantage because frozen warriors can't attack mages from a distance.

However, this does not guarantee that one class will always beat its weaker class. For example, a warrior can still defeat a lower levelled or worse-equipped mage, or a ranger can easily defeat a meleer who is stuck behind a fence.

Bending the triangle

RuneScape Combat Triangle Wallpaper — The Dwarves' "blueprint" of the Combat Triangle.

The triangle tends to focus mostly on the effectiveness of armour.

Warriors

Against Rangers. The warrior's armour is most effective against ranged projectiles, though the ranger has an advantage as long as they can remain out of direct contact.
Against Mages. The warrior's armour makes the mage's spells more potent. A warrior can take off metal armour or wear rangers' armour to raise magic defense; nevertheless, a mage can counter with binding spells.

Rangers

Against Warriors. The ranger can counter the warrior by evasion, or increase defence by switching to a chainbody.
Against Mages. The ranger's armour is most effective against magic, and holding tactics cannot prevent the ranger from hitting the mage. Therefore, this is a side of the triangle that is almost impossible to collapse.

Mages

Against Warriors. The mage's armour tends to offer little protection against warrior attacks. While the mage's attack hits easily through metal, warriors can still deal more damage in close combat. The mage can do little to change their position in the triangle, though members do have some more defensive armour options, such as splitbark. Also, members have access to the Ancient Magicks spellbook.
Against Rangers. The ranger's armour is most effective against magic, and holding tactics cannot prevent the ranger from hitting the mage. Therefore, this side of the triangle is almost impossible to collapse.
The prayer robes offers no protection or attacking bonuses although maces or staves if you are using Magic or Melee to help you in Prayer to help you inflict damage on your enemy. Explorer's ring also does this.

Hybrids are capable of standing at more than one point on the triangle, though at some cost and compromise to their equipment and inventory load.

Criticism

The combat triangle is highly criticised for being unbalanced, especially in favor of Melee combat and discouraging Mages.

Maximum Hits. This is often the subject of debate on forums, as the maximum hit of a mage is only 400 (with miasmic barrage, 370 with ice barrage). As of 12 February 2010 players may be able to hit well over 1000 with extremes, staff of light, hexcrest, and having ice Strykewyrms as a Slayer task (level 93 Slayer required). However, rangers can hit up to 1040 (520-520) if they utilise the Dark bow's special attack with Dragon arrows and high-grade ranger armour such as Void Knight equipment. Also, many melee weapons such as the Dragon dagger and Godswords can hit well over 700 with a high Strength level, and its maximum possible hit is currently shown to be 1,350 against kalphites using a Keris and the best strength equipment, the Piety prayer, a black mask or a slayer helmet, and drinking an extreme strength potion.
Prayer. Many prayers help warriors more than rangers or mages. Warriors can use Piety, Chivalry, or Turmoil to boost their stats, while rangers and mages are left with few others. The prayers that can be purchased through Dungeoneering, Rigour and Augury, are meant to act as the equivalent of Piety for rangers and mages - however, there is no ranged or magic equivelent of the curse Turmoil.

Warriors in Rangers' Armour. Another major criticism is the warrior's ability to wear Dragonhide armour, which does NOT reduce melee stats but increases magic defence, to fight mages with better protection than metal armour. This puts Mages at an even bigger disadvantage than the Warrior, since mage armour already often poorly protects against Melee weapons. Recent updates have improved the ability of mages to fight off these hybrid warriors, however. A mage can equip a slashing weapon to exploit the dragonhide armour's vulnerability to slash attacks. The Staff of light is not only a powerful staff, but also an excellent slashing weapon (however, meleeing with this requires hybridding); also, its special attack halves all melee damage dealt to the mage who casted it for one minute, drastically reducing the Warrior's damage output.
Relatively Low Defence of Mage Armour. Warrior armour provides the best protection against Melee and Ranged attacks. Ranger armour provides decent protection against Melee, Ranged, and Magic attacks. However, Magic armour often provides the least protection against all Melee and Ranged combat (the Infinity robes and Mystic Robes for example have no melee defense bonuses whatsoever), and comparatively little defence against magic attacks. Since mage robes are often cloth-based, Melee weapons can easily penetrate through them. Since mages are designed to be weaker than Rangers for the sake of "balance," mage robes are designed to provide little to no protection against Ranged attacks. And magic armour provide some defence against magic-based attacks, more than Melee armour but less than Ranger armour. This is consistently shown in Dungeoneering (mage robes of all tiers have around four times less melee defence than plate armour of the corresponding tier), minigames, God armour, etc.

Trivia

In the top right corner, among the Dwarven script, there are three equations that are used in game to calculate a player's combat level based on whichever combination of statistics results in the highest combat level.
The combat triangle is currently being rebalanced, due to complaints that mages and rangers are underpowered and warriors can easily kill all other classes on the combat triangle.
In the background of the Dwarf "blueprint", a hexagram can be seen.
It has been confirmed that when the combat triangle is rebalanced the dragon crossbow will be soon released but this could unbalance the combat triangle.
Dragonfire is another form of attack, and is not part of the combat triangle. It defies the combat triangle and cannot be blocked with any protection prayer. Using Protect from Magic will only partially help the user.
Damage inflicted by poison or a dwarven multicannon should also be considered another form of attack as Poison cannot be blocked by a protection prayer and a dwarven multicannon has its own symbol next to the damage inflicted by it showing that it does not inflict ranged damage but its own unique kind of attack instead. It is unknown if protect from range will defend a player from a cannon as it is impossible to expose a player to multicannonfire. However a monster with high range defence will be more difficult to hit with a multicannon and your range attack bonus does impact the multicannon's accuracy suggesting that multicannon damage and range damage are linked.
Although summoning familiars usually attack from a corner of the combat triangle the prayer protect from summoning will render you completely immune to their attacks even if multiple familiars are attacking you from multiple sides of the combat triangle.