When can I move my plants onto the balcony?

I'm very excited to grow a balcony garden this year, but I keep having to bring my sil boxes back inside because I'm afraid the wind and chill will kill them...I'm two floors up so there's no barricade from the wind for my little sprouts. What time of year do people normally start planting outdoors and putting out flower pots? Im in Toronto, and the temperature is ranging between 5-15 throughout this week. At what temp can I stop worrying and just plop them outside?