Yeast - fresh yeast (the best sort, IMO) can be obtained from any small baker (who bakes on the premises) or from a couple of supermarkets at the bakery counter:
Asda give it away
Morrison's will tell you it's in the chiller counters (it never is!), and,
Tesco's generally don't want to know!
However, all these supermarkets sell 125g of dried active yeast - Allinson's, in a yellow tin - for 64p, currently. For small batches of dough, use the same amount of dried as fresh - for larger amounts, use half the amount of dried to fresh.
Sachets of fast-action yeast can come in handy sometimes, but be aware it isn't 100% yeast, there are additives in there. Plus it's about 3 times the cost of the dried active. If you do need to buy some, get the own-brand version.
Flour. You'll get better results from strong, or bread flour, than you will from plain, although half and half works fine. I use own-brand white bread flour but I go for Doves organic wholemeal bread flour at £1.99 a bag. It's a very tasty flour.
Olive oil, if you can afford it, helps to improve the quality and keeping property of your bread. Lidl and Aldi basic brands score highly in tests and a 750g bottle will only set you back £2.20 or so and it lasts for ages.
I use basic ingredients - dried fruits, jam, cheapo grated cheese (I've never understood why this is cheaper than blocks of cheese - but it is!) and get fantastic results. Bread seems to bring out the best in other ingredients, somehow.
Maybe I'm biased! :)