This article is from the Recreational Figure Skating FAQ, by Karen Bryden with numerous contributions by others.
Chances are it's just a matter of confidence. You might be on your
inside edges because you have your feet spread apart, or because your
skates don't fit properly or are worn out.. When you're on an inside
edge, you've got your other foot to catch you. On an outside edge,
there's nothing between you and the ice but thin air.
Try one foot glides, straight at first and then on shallow outside
edges until you can securely glide for extended distances. Do them
near the boards at first if it helps to know there's something to grab
onto if needed. Like riding a bicycle, going faster will help you
balance. Be sure to practice on both feet equally.
Remember to keep your weight towards the rear of the blade, since
letting your weight shift forward puts you on the sharply curved,less
stable part of the blade profile.
Have the skates sharpened if necessary. You can't hold an edge on dull
skates. And have the skates checked to make sure the blades are