The magic utilized by our cast works by channeling the power of the very gods they inhabit. As such, it changes as these gods change and advance through the ages, and it changes as the domains of these gods intersect with each other. For the most part, only the magic that is native to the Library is relevant to our story, but the Museum and the Future both have unique magic systems of their own.
The foundational power of Library magic. All children of the Library have internal scrolls that detail the story of their life, and which grow the longer they live. These stories are the driving force of their life and memory. By tearing out part of the story you can erase someone's memory of those parts of their story. By rewriting it one could alter their memory, and by removing the story from their body you can effectively send someone into a coma that will only end upon reinsertion of the manuscript. In order to permanently kill someone, their story must be completely destroyed, though lesions and physical damage to their shell will effectively incapacitate them.
The expansion of one's story can, depending on whether or not they've been de-masked, strengthen their mask, and thusly their sense of identity, which results in a general increase in their magical power. However, the growth of one's story comes with its own negative effects. A growing story will eventually outgrow the body that contains it, damaging their shell and hindering physical capabilities.
Of note, the goddess of stories has power over all stories written in ink. She can, with this power, choose to enter and edit these stories, giving her the power to both alter one's memories and to control their present thoughts and actions.
Masks are associated with identity, so by wearing a mask one influences their sense of identity and the way they are perceived by others. The masks that the children of the Library are born with are physically attached to their stories, and can thus be influenced by the development of their stories. An individual that chooses to "demask" by physically tearing their story to detach the mask, may instead choose a face for themself, but these faces do not develop with their story. Demasking does pose some danger, as the process requires tearing one's story, which can cause memory loss, and temporarily requires one to go without a mask, which can physically erode their sense of personhood and identity if a suitable replacement is not found swiftly.
The default Blacksuit masks that all children of the Library are born with begin blank to reflect their intended use as mere tools and puppets, but, as they are connected to their manuscript, will slowly shift over time to reflect their developing identity. This takes the form of inked designs manifesting and "spreading" over the mask, typically starting out as simple geometric shapes that speak to a basic personality trait, such as a star representing someone who sticks out, or a square representing someone that is especially conformist. In later stages these shapes become larger and more intricate. The will of the Typewriter mandates that all shapes eventually develop into a ritualistic sigil enclosed by the Triangle of Solomon, representing how their power is supressed and controlled by a master. However, those who live longer may experience this mandated pattern beginning to expand and "repeat" until it covers their entire mask, this symbolic expansion beyond the control of their goddess poses a threat to her regime, and she thus goes to great measures to disempower the eldest Blacksuits.
The masks of the Red Gentry, being unconnected to their manuscripts, lack the ability to develop and grow more powerful with age, but can be empowered in a different way. Through the principle of conforminism, a mask designed after a particular character archetype can be imbued with powers associated with that character, which grow more powerful depending on how well an individual embodies the archetype they've chosen to represent themself by. The King's powers form a feedback loop, by subjugating more people he better embodies the archetype of the Ruler, which grants him the power to mentally influence and dominate others. The Inamorati can inspire love between others, though they cannot make it persist after the affected individuals have left their vicinity. At Il Dottore's peak he could heal people with a touch, and Dottore Peste has a paranormal proficiency in repairing the abstraction caused by contact with the Inbetween, and in teaching and communicating ideas to others. The Hunter's Ngontang mask grants him greater senses and proficiency in tracking things down.
Meanwhile, one who fails to embody their archetype may instead be plagued by the negative ptraits represented in their masks, such as the General, whose two-faced mask makes him paranoid and fearful of authority, instead of making him vigilant and heroic.
Very few beings are capable of perfectly embodying a mask, but some, specifically those born from tragedy or fear instead of being intentionally made by the Story Goddess, come quite close. The Collector, represented by a Hannya mask, encapsulates envy, and has the power to corrupt one's identity and eventually ingest it. Sepulchre wears an executioner's hood, and appears closer to a ghost than an individual, subtly influencing the environment around someone to manufacture their eventualy death, and being able to imbue fear of the end times in large numbers of people with relatively few words.
Not wearing a mask is often quick to result in someone losing their sense of personhood and identity, resulting in the loss of their magical power and motivation to pursue living. Those who choose to demask usually work to create a mask that represents the ideal person they want to become before actually demasking, in order to lessen the risk of identity loss.
Additionally, the powers represented by a mask are not just influenced by the physical appearance of the mask, but the materials used to create it. Wooden masks are more durable, but grant no additional magic, whilst masks made from the pages of stories inherit powers associated with that story. For example, the King may choose to make his mask from political documents and medieval romance literature to solidify his kingly powers, whilst the Doctors often make their masks from the pages of medical dramas and textbooks in order to grant them greater knowledge and understanding of these fields.
Objects are primarily used by the Blacksuits in order to substitute for their lack of unique masks to grant them personalized powers. Unlike masks which are manually made from wood or paper, magical objects are typically manifestations of stories, brought into existence through arcane reading rituals, or through the blessing of the Story Goddess. A reading ritual typically involves gathering several copies of the same book, tearing relevant passages out of each, and arranging them into a sort of summoning circle before reading out a detailed description of the item you'd like to manifest. The detail in your description is important, as a lack of it may result in the produced item being visibly "fuzzy" and generally less reliable. Items made by the Goddess's blessing require less work, and are simply written into existence by her decree, typically as a gift and sign of status to the higher-ups in the Blacksuits.
Magical objects display qualities in line with the symbolism associated with the object. For example, Murmur's Lantern of Truth grants him the power to see through lies and find his way when lost. Astaroth's Chekhov's Gun tends to grow more powerful the longer she stays in the background, allowing her to "charge" it by simply staying discreet for a time, before unleashing a shot that can blow straight through 6 feet of lead. Bifrons's keys are the most curious, granting him the ability to influence the topography of the Library, so that any door he goes through may relocate him to wherever he wishes to go.
Although most prominent among Blacksuits, certain members of the Red Gentry will also use this sort of magic, though their objects tend to be less reliable due to the human error involved in their making. The Martyr used Occam's Razor, causing those he cut to more easily be convinced of his lies, granted that the lies he spun required less steps to rationalize than the truth.
One of the central principles of Library magic, which can be seen in both how magic items and masks work. Throughout the history of art and literature humans have grouped and categorized characters and stories into different archetypes, tropes, and genres. Though the Library despises the constriction caused by this categorization, it can be quite useful to her inhabitants. By playing into these constraints, someone's magical power is greatly increased. This is seen in how an individual grows more powerful by playing into the archetype represented by their mask, but it is also relevant to the proper use of magic items, and can even play out on a mass scale. For example, the Library tries to organize the Blacksuits rigidly, by having them all in unison maintain the persona of dilligent worker bees in a great hive, they can become physically attuned to one another. The most dilligent Blacksuits can literally read each other's minds, and even less dilligent members, so long as they don't defect or otherwise openly defy the Library, have a mild ability to predict what their cohorts' next moves will be.
This is one of the principles that the King's plan is based on. By not just embracing the role of King himself, but by also having everyone else take on corresponding medieval roles, he hopes to become strong enough to begin literally restructuring the reality around him into his ideal paradise.
This principle tends to form something of a feedback loop. The more people the King subjects, the more kingly he becomes, allowing him to subject more people. Likewise, by gaining telepathy, the Blacksuits no longer need to speak, adding to the illusion that they are mere drones in service of a higher power, and thus expanding on their power. Unless something happens to deliberately break these loops, these factions could theoretically grow continually more powerful until they reach some sort of hard limit. Essentially, by properly embodying their archetype, their archetypical powers grow more powerful, which in turn allows them to better embody their role.
The Martyr has, reluctantly, embodied his archetype very well, to the point that his final death has inspired a real revolution. As a lingering effect of his magic, all of his followers' manuscrupts are tinted with his blood. This grants them immunity to the Typewriter altering their stories. Perhaps because of the nature of his archetype, his magic has become most powerful in his death. The strength of this magic is such that the Order of the Silence attempted to, and succeeded, in harnessing symbolism associated with him in order to bring him back to life by recreating the birth of Kvasir.
The Unwritten and the Inbetween are the last remnants of the old magic of the Library. As literary mediums have expanded to leave less room for interpretation, and as authorship has become a more accessible career, these domains have rapidly shrunk. Where once a play's script could be interpreted and performed in many different ways according to the whim of the director, books were released that tried to communicate a message less ambiguously. Now we have television, which doesn't even leave us with the freedom to imagine a character's appearance ourselves, this has shrunk the Inbetween's domain. Meanwhile, authorship has become a more accessible career as the advent of technology and the spread of literacy has made it easier for even amateur authors to write their ideas down on paper, shrinking the domain of the Unwritten.
Their magic works off the principle of uncertainty. Whilst out of view, creatures of the Unseen and Inbetween can do essentially anything with very few bounds. However, the moment they are brought into proper view their magic quickly constricts so that they can only use magic affiliated with their archetype. For more powerful entities of the Inbetween, such as godlike Eldritch horrors, this doesn't pose much of a problem, but for Unwritten entities of more mundane types, such as characters from mundane Slice of Life and Romance stories that were never published, this can leave them practically powerless.
The other interesting trait of Unwritten magic is how it interacts with the Written. When the creatures of the Inbetween interact with a Written entity, it causes them to be scrambled, their story rapidly shifting. The strength of this effect depends on the nature of the entity. Eldritch creatures corrupt their victims with incoherent ramblings about the true nature of the universe, often interspersed with Zalgo text. Meanwhile, a Mystery Man character may cause a mild corruption where one is mostly unaffected, with the exception that their story now tends to incorporate ominous phrases and prophecies.
Unwritten characters have a more ubiquitous effect, with their touch causing one's manuscript to "blank", words disappearing from the story as if erased, and in turn causing the Unwritten entity to become slightly more tangible as the stolen ink allows them to write themself into existence.