Lesson learned building mybirth.day: "take this memory off the wall" is one action with two meanings, and who taps it decides which.
If you wrote it, it goes back to being yours, filed nowhere. Not deleted: it stays in My memories as "Not on a wall". There's no restore behind that verb on purpose. Restore is for memories that were deleted, and it shouldn't be able to undo one you chose to take down. A memory lives on one wall, so it can't go back on that one.
If it's your wall and a friend wrote it, you're moderating. Nothing gets dropped. The memory is hidden, with when and by whom, and your friend sees "taken down" next to it in their own list. That way they can be told why it disappeared, and nobody gets quietly re-added and removed without a trace. Those two columns aren't fillable, so no form input can reach them by accident.
Either way, until you save your own arrangement the wall fills itself with the newest visible memories, so taking one off pulls the next one up.
mybirth.day/get
commit bddaa6a