I live in a Canadian English community on the border of Québec (the French-speaking province of Canada). Our population is bigger, we have more services, so a lot of the Anglophones barely speak any French at all. Most of the Francophones that I've encountered speak at least enough English to avail of these Anglophone services but some of them don't speak any English at all.
I'm not quite bilingual, but I'd consider my level of French to be above-average for Anglophones in my community. I can explain most things in French, I can understand French as long as it's not spoken quickly or with a lot of slang, and I could function if you plopped me in the middle of a Francophone country. However, I realise that my grammar is far from perfect, it's probably incredibly easy to tell that French is my second language, and I'm sure I have a noticeable English accent.
My question for you is -
If you were a Y-ophone who could speak a basic amount of X and you encounted an X-ophone who spoke a basic amount of Y, which language would you prefer speaking in?
Would it depend on your/your partner's grasp of their second-language? Would you appreciate it that the X-ophone is even attempting to speak Y to you? What would most likely happen?
My answer/opinion are in the comments. :)