This article is from the Problem Behaviors in Cats FAQ, posted to rec.pets.cats newsgroup. Maintained by Cindy Tittle Moore with numerous contributions by others.
Cats are very good at scaling fences. But if you have a yard that is otherwised fenced in, you can try keeping your cat from going over the fence by attaching corrougated fiberglass to the top of it. There is then no purchase for the cat to pull itself up. It is even possible to find different colors of the fiberglass to keep it inconspicuous. Keep in mind, though, that many cats are clever climbers and high jumpers and may circumvent anything short of a yard totally enclosed and roofed over with chicken wire.
You can try making an overhang on the top of the fence, if you cat cannot jump directly to the top of it. Use large bookshelf type angle brackets and drape netting or screening on it, to create an unstable barrier.
There are some "invisible fence" products for cats, where the perimiter of the yard is marked with a wire that will activate an electric collar on the the cat. Do not use these without supervision, and ideally they should be used in conjunction with a visible fence that the cat can use as a visual reminder of its constraints. This seems to work well with some cats and not at all with others.