Balancing Encounters in D&D

Dungeons and Dragons 5th Edition can be notoriously hard when it comes to balancing encounters. While not every fight has be a bare knuckle, near death experience, when it comes time to design a difficult combat it can be hard to get things right, especially as levels increase. Many DM's find the party unexpectedly trouncing their Big Bad epic, end of story arc boss fight in two rounds without breaking a sweat. Or being completely overmastered as the DM desperately fudges to keep them alive. Both are  unsatisfying experiences.

The CR system that 5e provides is at best a rough guide and at worst a completely false sense of security that will utterly betray you.  So how should you balance one of these ultimate encounters?

The Goals of having a combat

First, figure out the type of fight you are designing. Combat can serve many purposes, but it is almost always about eliciting an emotional responses in the parts of the players. The most common emotional arc is feelings of tension and danger followed by release with the associated rush of heroism and triumph.

Heroic Triumph

Catering to heroism is the easiest. A sharp brawl, where the party gets bloodied but overcomes the enemy provides a rush of accomplishment. It's fun to be powerful, and it's good practice for more serious confrontations. Fight's don't always need to be deadly.

Building Tension

Ratchetting up the tension is another common arc for combat, as the DM demonstrates to the party they are under pressure, under attack, and that their resources are becoming depleted. Again, the battle doesn't have to be a gut wrenching life or death struggle, but the conflict needs to be persistent and unending. Each hitpoint lost, each spell expended needs to give the players the impression they are being worn down. This technique can be especially effective if it is combined with some story driven mechanism to make long rests difficult. Deprive them of rest and harry them and your players will start to feel the pressure.

Both of these feelings can also be easily combined. If the fight is hard enough to wear the party down, but still not so hard that it's a real struggle, then both emotions combine. "We are bad asses in a tough spot."

Designing Epic Encounters        

Sometimes things need to feel serious. Climatic. Dangerous. I define Epic Encounters as "encounters that present a real risk of a TPK." Usually I don't expect a TPK unless someone is exceptionably stupid somewhere along the line, or the party as a whole performs poorly tactically. But I do make sure the party realizes that this risk DOES exist. That this fight could be The End.

"Someone is exceptionably stupid" also has a corollary. If the party is exceptionably smart and well organized leading up to this battle (say uncovering an enemies weakness, enlisting allies, etc) the DM should allow this epic encounter to be "easier". If you do this and continually refer to it (Boy you are sure are lucky you learned the BBEG's weakness to cold and then quested for cold damaging items) it can actually make the battle even more satisfying then one where the BBEG is overcome by sheer force. 

Both these things aside, assuming an average level of preparatory performance, you want a fight that will be decided by good tactical play

Designing such an encounter is hard in 5E. You can't rely on CR. So what do you rely on?

Some factors for balancing encounters

1. Be aware of your parties damage potential, how much damage your party can put out in a round. Calculate it over several rounds, assuming they pull out all stops. Whatever creature you are using, adjust hitpoints accordingly so they don't die in the first three rounds. That’s the single most important factor in scaling IMO. I pretty much always beef up hit points for parties over 4 in size.
2. The second is the action economy. The bad guys side needs to be acting at least half as much as the party (preferably as often as the party) or else the party will control the battlefield and min/max the combat by controlling who can be attacked. You can add actions via adding minions, special attacks and legendary actions. I will often beef up legendary actions from the stock monsters.
3. The third is balancing burst damage vs sustained damage  from monsters. Sustained damage is almost a spreadsheet in it's predictability and less exciting, it's the foundation of the house but not the house. Burst damage upfront will make the payers feel this is a serious challenge and force some tactical adjustment, while sustained damage should be enough to keep the heat on. A great example is a dragon, he hits you hard with his breath weapon, probably on turn 1 and then falls back on his teeth and claws. Burst damage later in the combat can be an unexpected and fear inducing surprise (like the hydra). Another form of burst damage can be simple reinforcements.
4. Make your monsters mobile. A slow monster is a dead monster. Give them extra movement as part of legendary actions. Give them abilities that interfere with or cancel out PC movement options. Don’t forget about the disengage action. Keeping the smart ones off the squishier casters should be a major challenge
5. Give your monsters a mix of skills. Much has been said in the halls of D&D about the value of a combined arms approach, so I won't belabor it, but a group of enemy that have a mix of magics, ranged and front line melee will generally outperform a homogenous group.
6. If you want your party to perform well tactically, you need to give them an interesting tactical environment to perform in. Terrain and terrain features are probably the most important feature here, environmental effects and hazards also matter. This can also be combined with good prep, a party that is clever or skilled before the battle should have some control over the terrain the battle is fought over. 

There are many more tips to a good encounter of course, but this is a reasonable start.