I've tried this method, one with a native speaker, and again with a fellow learner. While I can't argue that it's probably the best way learn it does offer some hardships.
For example, when conversing with the native, I ran through everything I had learned; genki, interests, hobbies, weather, and so on. Then I ran out of things to say and the conversation stalled. So then we started chatting in english since it seems (to me) that everyone in japan knows english better than I do. The fellow learner was even worse. We both kept trying to make up sentences using english phrases and idioms, and the resulting lines made no sense to anyone. Not even us who had written them.
I would say that the best way to use this practice is after you spend a few weeks (months? years? decades?) getting used to basic grammar, and building your vocabulary. Otherwise you end up saying the same three sentences and never get any further.
That is a very good point.
Before I started doing private lesons with my teacher I had studied in a school for about 6 months (attending twice a week) and I had just passed the level 4 test. However, I did not really speak that much Japanese.
In every lesson we study using a textbook, so we have something to do when we run out of conversation. It is hard work getting through those early lessons when you can`t really express what you want. However, after time it starts to get more fun as you can express more.