If you are planning to visit the park for 2-3 days, it won't matter too much when you go as that is enough time to do everything even on a holiday weekend. If you're looking for the least crowded time, the earlier in June or later in August the better. What days are best to go, however, are dependent on more factors. If you just want to visit the park, go when it's least crowded. If you want to be sure everything will be open, however, you need to wait until at least mid-late June as sometimes SFMM's off-season maintenance runs long. Finally, if you count credits and want to make sure you can ride SFMM's new family coaster, don't go before 4th of July.
I still think you should go in early/mid June or mid/late August on weekdays. If these don't work, any other time except right around 4th of July should be about the same. Just avoid weekends and you'll be fine, especially if you have multiple days, as one full day is sufficient for the park anytime except a holiday weekend.
Walt Disney World
Tokyo Disney Resort
Theme Park Insider Books