I love to use non-English languages in a story. To me, a foreign language behaves like a form of music - it creates mood and atmosphere for the readers who don't understand it, and it further creates a layer of meaning for those readers who do understand it. I take much the same philosophy toward foreign language as I do toward alien language: the more you use, the more alien the story feels; the less you use, the more familiar the language feels.
With a story like "Suteta Mono de wa Nai (Not Easily Thrown Away)," it's really important to me to have a point of view that suggests a person who is inside the Japanese perspective. That means using a lot of English, which might seem ironic. Making a mostly English narrative feel Japanese is all about exploring the other aspects of the Japanese point of view, like metaphors, attitudes, culturally grounded phrases and opinions, all of which can be accomplished in English. That English narrative creates a scaffold into which Japanese phrases can be embedded without having to be defined.
When the context surrounding a word is strong, the word does not need translation.
But what constitutes a strong context differs for different people. I, personally, could totally fail to provide any scaffolding context at all and still appreciate what those words are doing, which means I have to guess in my first draft which ones will be easily deduced by others and which ones won't.
Also, it depends on who is reading it.
Say, for example, with the identity of Naoko's grandmother, also known as Obaa-chan. A lot of people do know that phrase; a lot don't. In this instance I relied on Grice's Cooperative Principle of Communication, the Maxim of Quantity which says "say as much as, and no more than, is required." At the start of the story, there is one character hearing voices. The voices mention she has a grandmother. Then the single character refers to someone named Obaa-chan. Conversational logic tells us that Obaa-chan has to be her grandmother, because only one other flesh-and-blood character has been mentioned, and that's her grandmother. Right? Right.
But it's not that obvious for everyone (at least one of my readers mentioned feeling this way).
Just because something makes sense to you, that doesn't mean you can count on every reader to put together the same logic. Conversational logic depends a lot on our previous knowledge and experience!
It's just like when I hear someone saying a sf/f character name that I wrote down, and saying it in a way I consider "wrong." Who's to say that's wrong? It's what was on the page, filtered through their own phonological interpretation of the spelling. I decided long ago that it was up to them how to say it and I could explain the "real" pronunciation if asked, but didn't really have to.
When you are working with an alien language that nobody knows, you can be certain that nobody knows it. But when you are working with a foreign language, it's a different kind of balancing act. Some people know it, and some don't. More importantly, these people come to a story with different attitudes.
Some people are content to let foreign language act like musical accompaniment, and some people feel like they want to know the meaning of every word. The writer is usually standing somewhere in between.
In my own early drafts of "Suteta Mono de wa Nai (Not Easily Thrown Away)," I discovered something I hadn't expected. A reader with no knowledge of Japanese at all was perfectly happy to let the Japanese function as musical accompaniment so long as the English parts of the story made sense, and deduction could reconstruct the meanings in critical spots. A reader with lots of knowledge of Japanese was content to see a level of redundancy in the piece because they knew that not everyone would know all the words they did. A reader with some knowledge of Japanese, but not enough to understand all the words used, was discontent with not knowing all those words and wanted to see stronger scaffolding.
Once you move out into the world where a large number of people are reading, this selection of reactions diversifies. I've heard the no-Japanese reaction a lot, I've heard the lots-of-Japanese reaction a lot, and I've heard multiple different versions of the some-Japanese reaction. My sense is that a person's reaction will be very individual, not only dependent on how much Japanese they know, but how much they want to know, how much they personally expect others to know, etc. This will be similar (with some variations) no matter what language you choose to use.
This is all part of sending a story out into the world. Every reader reads a story differently. Every reader brings different expectations. So how much Japanese is too much? How much scaffolding is too much? It's a tricky balancing act. As a writer, all you can do is respond to critique to a level that seems reasonable based on what you know about your beta readers, and then trust that later readers will read it in their own way. Because they will - that's what reading is.
It's something to think about.