Herb gardens are great. They allow you to be spontaneous in your cooking while keeping things fresh. I've slowly been growing mine over the past couple years. It's a good way to test the waters with your gardening skills. It is also inexpensive.
I've been storing mine in vintage milk glass (and just white) containers I've been finding from the thrift store and estate sales. I'd highly recommend it because:
1. They all go together, but you don't have to worry about them matching perfectly.
2. You can add more at your own pace.
3. They're less expensive than buying containers from the store.
4. They're easy to find. They're are way too many candy dishes out there, lets try to fill them with something more healthy.
5. They're heavy enough to not get tipped over by the wind and animals etc.
Here is where I would recommend getting plants:
1. A dollar store: they have plants that have already been started. I found arugula, parsley, and cilantro there.
2. Lowe's: depending on the time of year, you can get plants there for the same price as they would be in the grocery store, and this way you don't have to worry about them going bad in the fridge.
3. Trader Joe's: They often have herbs in the plant section. I got my rosemary, basil, and sage from there simply because I was already getting groceries and it was convenient.
4. Grow them from seed: I'll be honest, I've heard great things, and yes it is the least expensive, but I've never had the patience for this. I'll save that for growing tomatoes and peppers.