The Lee 50th Anniversary Breech Lock Challenger kit as mentioned above is an excellent value. Hard to beat $90 for what it includes:
MidwayUSA - Lee Challenger Breech Lock Single Stage Press Anniversary Kit
It will do rifle and handgun, and a single stage is an excellent press to learn on and always useful even if you eventually get a turret or progressive setup. I loaded on a single stage for about seven years before I got any sort of faster system, and I still do about half of all my handloading on a couple of different single stages (a RCBS Rockchucker II and a Lee Breech Lock Challenger).
With the kit listed above you can get started with the addition of a manual or three (the Lee manual is perfectly good and inexpensive), a set of calipers, dies, a bullet puller (for mistakes), and components. You will eventually want to add additional items, but the above will get you going.
A kit like this:
MidwayUSA - RCBS Rock Chucker Supreme Single Stage Press Master Kit
Will get you some much more refined equipment, but it costs $200 more than the Lee kit and in my experience doesn't make better ammunition.
As I said, I own both the Rockchucker and the Breech Lock Challenger. The Rockchucker is clearly a more robust press, I do my rifle resizing on it (not the the Lee couldn't handle it, but the RC is built for that sort of task). However, the Challenger has a much better priming system (the Safety Prime) than the RC, and the Challenger's primer catcher is a lot more reliable. Primers tend to bounce out of the RC's catcher and go everywhere. The huge majority of the spent primers actually go where they are supposed to with the Lee. So, I always decap and prime on the Challenger when I'm loading on single stages.