Good branching dialogue is hard to do, and I see a lot of writers (even very experienced ones) struggle to get it right. I don't claim to have discovered one perfect technique to getting it right, but I find the following guidelines can help avoid some of the worst pitfalls.
1. Know your player. This is surely the most important and also one of the more challenging aspects of writing branching dialogue. One of your characters in your scene is necessarily in the player's control, which means they are at least partially out of yours. It might seem like stating the obvious, but I have seen many branching dialogue scenes in which the writer seemed unwilling to relinquish control of either character.
But the thing is, the only reason to even bother with branching narrative is because the player is there. A single isolated path of branching narrative is never going to be better than a well-crafted linear story from a pure narrative flow and structure standpoint. We do it because it adds value to the player's experience by letting them be part of it. So forget about your perfect story.
What is the player here for?
Let's take Mass Effect as a quick example. Why is it so beloved? It knows what the players are there for. To be the space marine action hero, in a variety of flavors, but mostly the true-hearted hero who is loyal to her friends and always finds the way to save the galaxy. And, of course, to romance aliens. So even, and especially, when a choice yields little to no consequences in Mass Effect, the player nevertheless gets a payoff, because the game let them play their flavor of that hero.
Now, not every game is Mass Effect. Figure out what your players want, what their core fantasy is, and let them play those characters.
2. React to the player. Sensing a theme here? Video games at their most fundamental level are a series of actions & reactions, inputs and outputs. It's a given in other aspects of game design that feedback is essential to giving the player a great experience. Designers, artists, and programmers spend months and years getting "gun feel" just right, so that players can easily interpret when they hit, when they miss, when they need to reload, and so that it feels great doing all of those things.
Branching dialogue is similar. You do not need a huge, complicated branch to get this. Just make your choices feel great to choose. Make them sufficiently distinct, sufficiently clever, and most of all, make the reactions of the other characters worthwhile. There's nothing worse than picking a line and having an NPC say the equivalent of "Okay, but as I was saying ..."
3. Give the player just enough info that they think they know what they're doing. You don't need to spell out what the consequences of each choice will be, but choices always feel better to players if they believe they have enough info to weigh one against the other. They should always have some basis for choosing, whether it's their understanding of the character they're interacting with, the worldbuilding, or their own sense of right and wrong.
I like how Overboard! has done this recently. I know that as Veronica I have killed my husband and want to get away with it. My first choice is how to respond to the steward knocking on my door. I don't know exactly what the consequences of different responses will be, but I know enough (that I want to appear innocent and get away with murder) to make a choice and see what comes of it. Of course, Overboard! has the advantage of being designed for replay, so not fully anticipating the consequences of a choice is less punishing. Most branching dialogue is, unfortunately, not replayed.
4. Surprise the player! You don't want to pull the rug out from under them entirely (that's called lying, and telling stories is an exercise in trust), but it's always memorable if things turned out almost as planned, except for one little wrinkle, complication, or surprising development.
5. Be wary of scope. How drastic a branch you can execute will depend greatly on the type of game you're building, but I see a lot of branching samples that build to one massive, game-altering choice and my first question is always, "How are you going to write your way out of this one?"
Choices that feel massive to the player should produce massive consequences, otherwise, they undermine the player's faith in all of your choices, large and small, to have an impact (again, telling stories, trust.) But massive consequences often lead to double the writing, double the levels, double the combat encounters, etc., and most companies are not willing to invest in entire levels that half the playerbase (or less: big, binary choice paths rarely split the audience 50-50) will never see.
Get good at having big choices that impact character relationships, or that are limited to a region of the game that the player moves on from, and please avoid writing choices that amount to A: the player goes on the quest; B: the player retires to a quiet life on a farm somewhere. Your job is to come up with compelling reasons for the player to go on the adventure, not to invite them to opt out of it the first chance they get.
As a player, I really love it when I get to be part of a game's story, and I hope that these tips will set you on the path to writing better, more rewarding branching dialogue for your players.