The only custom book games I've played are on OCTGN. All of my IRL games have been in apprentice mode. Most of those I've used the spellbooks for.
I think the important thing here is to be able to find your cards when you need them. If you're using the binder, then it's probably good enough to organize by type.
On OCTGN everything is organized by type for you. It's also spread out in a row in front of you so that you can easily see all your cards, in a way that would be impossible to hide them from your opponent(s) if it were in IRL.
However, once I do get around to playing custom games IRL, I'm planning on playing with my spellbook in deck form. It will start the game organized by type, then strategic relevance.
However, during the game, I will actively keep my deck organized by type, but not strategic relevance.
For those who have trouble finding their spells in a deck, my recommendation is the more organized your deck is at the start of the game, the easier it will probably be to find the cards you need when you need them. It might be good to organize first by type, then subtype, and then level. Or you could skip subtype and organize by type then level.
However, I would also advise players not to try too hard to maintain this level of organization throughout the game. If it's that well organized at the start then I think it's probably good enough to maintain organization by type.