From 38d9ffacc58ac2254d7ec8aad502a3f9d76d08f7 Mon Sep 17 00:00:00 2001 From: Jean-Pierre Martineau Date: Wed, 16 Sep 2026 20:29:02 -0400 Subject: [PATCH] Every glyph that may be drawn as an emoji is reserved two cells The curated narrow and wide sets are gone, with the probe that decided when to apply them and the add-narrow-emoji skill that fed them. They lined up the rows somebody had already looked at and none of the rows nobody had, and a contact's name arrives off the air carrying whatever its owner typed. In their place, one structural rule: a glyph built from several codepoints (a selector sequence, a ZWJ sequence, a keycap, a flag) or a lone text-default emoji or Regional Indicator is reserved two cells, the most a single glyph draws. Pinned into two cells, a glyph the font draws in two fills them and one it draws in one leaves a blank beside it, so the text after any emoji starts in the same column on every row. Pinning alone could only hold the lanes after it. Two exceptions keep their one cell: Rich's single-cell ranges, which its fast path measures without asking, and the nine text-default emoji the app draws itself as marks. That list is closed by a test over the package's source, so a new mark cannot slip into the reserve unnoticed. Rich measures in three places and prompt_toolkit in a fourth, and all four are patched from the same rule at once: Rich's segment splitter walks a string until a whole-string width meets its cut, and a character measured differently from the string around it is a cut that walk never meets. Every multi-codepoint glyph is also merged into one fragment before layout, so a mark followed by a selector is not folded into a one-cell slot. Installed only by the full-screen app on a platform that draws emoji. The command line keeps the stock tables, because nothing pins its output. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/add-narrow-emoji/SKILL.md | 107 --- .claude/skills/add-narrow-emoji/classify.py | 144 ---- meshterm/platforms.py | 9 +- meshterm/ui/menu.py | 14 +- meshterm/ui/tui/colsnap.py | 31 +- meshterm/ui/tui/emoji_width.py | 903 +++++++------------- tests/test_emoji_width.py | 617 +++++++------ tests/test_win32dll.py | 16 +- 8 files changed, 622 insertions(+), 1219 deletions(-) delete mode 100644 .claude/skills/add-narrow-emoji/SKILL.md delete mode 100644 .claude/skills/add-narrow-emoji/classify.py diff --git a/.claude/skills/add-narrow-emoji/SKILL.md b/.claude/skills/add-narrow-emoji/SKILL.md deleted file mode 100644 index 0af8900..0000000 --- a/.claude/skills/add-narrow-emoji/SKILL.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -name: add-narrow-emoji -description: Fix a MeshTerm row whose layout an emoji breaks — a right border pulled a column short, or a glyph overrunning its lane — because the terminal draws it in a different number of cells than Rich/prompt_toolkit measure. Works in both directions: a glyph drawn narrower than measured joins the narrow allowlist, one drawn wider joins the wide set. Classifies the emoji (lone codepoint, country flag, VS16 sequence, or ZWJ/other cluster), says whether it is already listed, and adds it the correct way. Use whenever an emoji "fucks up the layout", notches a panel border, or smears a row. ---- - -# Adding an emoji to MeshTerm's terminal-aligned width sets - -MeshTerm aligns Rich's and prompt_toolkit's glyph widths with how the terminal -*actually draws* an emoji, so a row frames flush instead of notching its right border a -column short or overrunning its lane. The machinery lives in -[emoji_width.py](../../..//meshterm/ui/tui/emoji_width.py); the design is in that module's -docstring and in the `glyph-width-alignment` memory. Background: the terminal's -narrow/wide split is **per-glyph and font-driven** — there is no rule and no way to probe -it (a cursor probe reads the PTY, not the renderer) — so the exceptions are two -hand-curated sets, extended one confirmed glyph at a time. This skill does that one -addition correctly. - -## When to run - -An emoji in a row breaks its layout, in either of two ways: - -- **Border pulled in** (a blank sits between the border and the frame edge), or a smear to - the right of the glyph — the terminal draws it in *fewer* cells than the authorities - reserve. → the narrow allowlist (`_DEFAULT_NARROW_LONE`). -- **The row overruns**, pushing what follows a column right — the terminal draws it in - *more* cells than the authorities reserve. → the wide set (`_DEFAULT_WIDE_BASE`). In - practice this only happens to a **VS16 sequence** the narrow verdict got wrong; a *bare* - codepoint both authorities call one cell is measured correctly and must be left alone. - -Step 1 tells you which; it also tells you when the glyph is **already listed**, which is -the commonest answer to "please add this one". - -## Steps - -1. **Classify the emoji.** Run, from the repo root: - - The interpreter below is `$DEV_VENV_PYTHON` from `.dev.env` at the repo root (gitignored; -`.dev.env.example` shows the shape) — read it first. - -``` - $DEV_VENV_PYTHON .claude/skills/add-narrow-emoji/classify.py - ``` - - It prints the codepoints, how each authority measures it now (`Rich now`, `pt now`) - and after the fix (`Rich fixed`, `pt fixed`), the class, and the action. `pt now` is - the number that matters — prompt_toolkit places the panel border. - -2. **Act on the class.** The machinery runs in *both* directions — a glyph drawn narrower - than measured joins `_DEFAULT_NARROW_LONE`, one drawn wider joins `_DEFAULT_WIDE_BASE` - — and the classifier names which, including "already listed", the commonest answer: - - **already listed** (either set): nothing to add. If a row still misaligns, some - *other* glyph on it is the culprit — classify that one. - - **lone wide codepoint** (e.g. `👋`, `👍`): the narrow-allowlist case. Only add it **if - the terminal draws it in one cell** — confirm visually first. Append the character to - `_DEFAULT_NARROW_LONE` in [emoji_width.py](../../..//meshterm/ui/tui/emoji_width.py): - `_DEFAULT_NARROW_LONE = "👋👍"`. (For a throwaway session test instead of a code - change, set `MESHTERM_NARROW_EMOJI="👋👍"` in the environment.) - - **lone narrow codepoint, bare** (e.g. `🛣` U+1F6E3, `🕸` U+1F578): **no change**, and - resist the pull to "fix" it. Both authorities say one because the codepoint is outside - Emoji_Presentation, and with no VS16 asking for emoji presentation the font draws a - one-cell *text* glyph — they are right. Forcing it into `_DEFAULT_WIDE_BASE` reserves a - cell the terminal never draws and pulls the row's border a column **in**; that is - exactly what listing `🛣` did to the Trophy case, while `🕸` — same class, one board - down, never listed — framed flush throughout. Both authorities already agreeing is not - a bug to correct. - - **VS16 emoji-presentation sequence** (e.g. `☀️`): **handled by default** (the variation - selector is skipped, so the base is measured alone). But the renderer is not uniform — - if *this* one paints two cells wide, add its **base** codepoint to `_DEFAULT_WIDE_BASE`, - as `🛩️` (U+1F6E9) needed. Keying on the base makes the bare form ride along. - - **flag (Regional Indicator pair)** (e.g. `🇨🇦`): **already handled** as a whole - category — no entry. Confirm `pt fixed` is 2 and stop. Never list an individual flag: - indicators are shared, so it would half-fix that flag's neighbours and skip the rest. - - **ZWJ sequence / keycap / skin-tone / other cluster** (e.g. `👨‍👩‍👧`, `1️⃣`, `👋🏽`): - **not covered** by the lone/flag mechanism — resizing its parts individually would - misalign it. Do not force it into either set; note it for bespoke handling. - -3. **Extend the tests.** In [tests/test_emoji_width.py](../../..//tests/test_emoji_width.py), - add the new glyph to `test_rich_cell_len_narrows_only_allowlisted_lone_emoji` and - `test_pt_cache_narrows_only_allowlisted_lone_emoji` (both authorities → one cell for a - narrow addition, two for a wide one), and to the width-1 calibrate test. Keep a - known-wide emoji (`📡`) asserted at two, so a future over-broad change is caught. - -4. **Run the tests:** - - ``` - $DEV_VENV_PYTHON -m pytest tests/test_emoji_width.py -q - ``` - -5. **Confirm visually.** The fix can't be reproduced headlessly — a non-interactive run - keeps Rich's defaults, and CI can't see the renderer. Ask the person on the affected - terminal to send a chat message with the glyph and confirm the border is now flush and - nothing shifted. Only then is the addition trustworthy. - -6. **Commit** with a message naming the glyph and why (see the repo's commit style). - -## Guardrails - -- **Never narrow a glyph the terminal draws two wide** (a menu icon like `📡`, `💬`), and - never widen one it draws in one cell. Each set only helps glyphs mismeasured in its own - direction; a wrong entry *introduces* the misalignment it was meant to remove. When - unsure, leave it out. -- **One glyph, confirmed, at a time.** An unlisted emoji is no worse than before (the - `erase_down` full-repaint still prevents a narrow one's smear); a wrongly-listed one - breaks a border. Bias to omission. -- **Do not touch Rich/prompt_toolkit width globally.** A blanket emoji remap has been tried - and reverted twice — it wrecks the menu icons the terminal genuinely draws two wide. The - curated allowlist and the flag category are the only sanctioned levers. diff --git a/.claude/skills/add-narrow-emoji/classify.py b/.claude/skills/add-narrow-emoji/classify.py deleted file mode 100644 index e82502b..0000000 --- a/.claude/skills/add-narrow-emoji/classify.py +++ /dev/null @@ -1,144 +0,0 @@ -r"""Diagnose one emoji for the MeshTerm terminal-aligned width machinery. - -Given an emoji on the command line, report its codepoints and how the three width -authorities measure it (wcwidth, Rich, prompt_toolkit), then classify it and print the -exact action the `add-narrow-emoji` skill should take. Run from the repo root: - - $DEV_VENV_PYTHON .claude/skills/add-narrow-emoji/classify.py 👋 - -(``$DEV_VENV_PYTHON`` is the interpreter named in ``.dev.env``; ``.dev.env.example`` shows -the shape.) - -The point is to remove the guesswork the skill's judgement call depends on: a lone wide -codepoint gets an allowlist entry, a flag is already handled by category, and a ZWJ / -keycap / modifier cluster is *not* covered and must not be shoehorned in. -""" - -from __future__ import annotations - -import pathlib -import sys - -# The chat border fix only ever engages on a terminal that draws emoji narrow; measure -# there or trust the maintainer's eyes. This script just classifies — it never patches. -sys.stdout.reconfigure(encoding="utf-8") # print emoji safely on a cp1252 console -# Run from anywhere: put the repo root (four levels up) on the path so `meshterm` imports. -sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[3])) - -import rich.cells as rc # noqa: E402 -from prompt_toolkit.utils import get_cwidth # noqa: E402 -from wcwidth import wcwidth # noqa: E402 - -from meshterm.ui.tui.emoji_width import ( # noqa: E402 - _DEFAULT_NARROW_LONE, - _DEFAULT_WIDE_BASE, - _is_regional_indicator, - _make_cell_len, - _make_pt_cache, -) - -_ZWJ = "‍" -_VS16 = "️" - -_NARROW = frozenset(_DEFAULT_NARROW_LONE) -_WIDE = frozenset(_DEFAULT_WIDE_BASE) - frozenset((_ZWJ, _VS16)) - - -def _base(emoji: str) -> str: - """The emoji's base codepoint — what the wide set is keyed on (selectors dropped).""" - others = [c for c in emoji if c not in (_VS16, _ZWJ)] - return others[0] if len(others) == 1 else "" - - -def _classify(emoji: str) -> tuple[str, str]: - """Return ``(label, action)`` for ``emoji`` — what it is and what the skill should do.""" - if not emoji: - return "empty", "Nothing to classify — pass an emoji as the argument." - - # Already curated? Say so first — it is the commonest answer to "please add this one", - # and every branch below would otherwise re-derive advice for a decision already made. - base = _base(emoji) - if base and base in _WIDE: - return ( - "lone codepoint drawn WIDE — already listed", - "ALREADY IN _DEFAULT_WIDE_BASE (both authorities measure it 2, bare and with its " - "VS16 selector). Nothing to add. If a row still misaligns, the culprit is another " - "glyph on it — classify that one instead.", - ) - if len(emoji) == 1 and emoji in _NARROW: - return ( - "lone codepoint drawn NARROW — already listed", - "ALREADY IN _DEFAULT_NARROW_LONE (both authorities measure it 1). Nothing to add.", - ) - - if len(emoji) == 2 and all(_is_regional_indicator(c) for c in emoji): - return ( - "flag (Regional Indicator pair)", - "ALREADY HANDLED as a category — no allowlist entry. Verify pt width is 2 below; " - "if so there is nothing to add. Never list a single flag (it half-fixes its " - "neighbours that share an indicator).", - ) - if _ZWJ in emoji: - return ( - "ZWJ sequence (multi-glyph cluster)", - "NOT covered by the lone/flag mechanism (a family, a profession, a flag-with-tag). " - "Do not add it to either set — the parts would be resized individually and " - "misalign. Flag this to a maintainer for bespoke handling.", - ) - if len(emoji) == 1: - if wcwidth(emoji) == 2: - return ( - "lone wide codepoint", - "IF your terminal draws it in ONE cell, add it to _DEFAULT_NARROW_LONE in " - "meshterm/ui/tui/emoji_width.py (or MESHTERM_NARROW_EMOJI for a session test). " - "If your terminal draws it TWO wide, leave it — narrowing would break its border.", - ) - return ( - "lone narrow codepoint (bare, text presentation)", - "NO CHANGE — and resist the pull to 'fix' it. Both authorities measure it one " - "because it sits outside Emoji_Presentation, and with no VS16 asking for emoji " - "presentation the font draws a one-cell text glyph: they are right. Forcing it " - "to two reserves a cell the terminal never draws and pulls the row's border a " - "column IN — the mistake U+1F6E3 🛣 cost the Trophy case. If this row really is " - "misaligned, another glyph on it is the culprit; classify that one.", - ) - if base and _VS16 in emoji: - return ( - "VS16 emoji-presentation sequence", - "HANDLED BY DEFAULT (the variation selector is skipped, so the base is measured " - "alone and the narrow-VS16 verdict applies). If your terminal draws THIS one two " - "cells wide anyway — the renderer is not uniform — add its BASE codepoint to " - "_DEFAULT_WIDE_BASE (or MESHTERM_WIDE_EMOJI), as U+1F6E9 🛩️ needed.", - ) - return ( - "multi-codepoint cluster (keycap / skin-tone / other)", - "NOT a plain lone codepoint — the lone/flag mechanism does not cover it. Flag this to a " - "maintainer rather than adding parts to either set.", - ) - - -def main(argv: list[str]) -> int: - if len(argv) != 2: - print("usage: classify.py ") - return 2 - emoji = argv[1] - label, action = _classify(emoji) - - print(f"emoji : {emoji}") - print("codepoints : " + " ".join(f"U+{ord(c):04X}" for c in emoji)) - print(f"grapheme : {len(emoji)} codepoint(s)") - print(f"wcwidth : {sum(max(0, wcwidth(c)) for c in emoji)} (per char: " - + ", ".join(str(wcwidth(c)) for c in emoji) + ")") - print(f"Rich now : {rc.cell_len(emoji)}") - print(f"pt now : {get_cwidth(emoji)} <- the authority that places the panel border") - # "fixed" = what the width-1 calibration path already produces today, both sets applied. - print(f"Rich fixed : {_make_cell_len(_NARROW, _WIDE)(emoji)}") - print(f"pt fixed : {_make_pt_cache(_NARROW, _WIDE)[emoji]}") - print() - print(f"class : {label}") - print(f"action : {action}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main(sys.argv)) diff --git a/meshterm/platforms.py b/meshterm/platforms.py index 1c29ec3..958f5f5 100644 --- a/meshterm/platforms.py +++ b/meshterm/platforms.py @@ -75,10 +75,11 @@ class Platform: this is ``False`` there. ``MESHTERM_FULL_WIDTH`` remains an explicit override on top of this default. emoji: Whether emoji icons render at all. ``False`` means every icon funnel routes - through a compact single-BMP-glyph table instead (P3), and the emoji-width - calibration probe (:func:`~meshterm.ui.tui.emoji_width.calibrate`) is skipped - outright — there is nothing to calibrate once no emoji are ever drawn, and the - probe writes escape sequences the console font can't shape anyway. + through a compact single-BMP-glyph table instead (P3), and neither half of the + emoji alignment runs — the reserve-two measurement + (:func:`~meshterm.ui.tui.emoji_width.install`) or the column pinning + (:mod:`~meshterm.ui.tui.colsnap`). Once no emoji are ever drawn, every glyph on + screen is one the console font has verified, and the stock widths are exact. ascii_fold: Whether names and message bodies are NFKD-folded (accents stripped) at the render boundary before display. Storage is never touched. Wired in P3. truecolor: Whether the theme may use arbitrary 24-bit SGR colour. ``False`` selects diff --git a/meshterm/ui/menu.py b/meshterm/ui/menu.py index 789d649..6588850 100644 --- a/meshterm/ui/menu.py +++ b/meshterm/ui/menu.py @@ -48,7 +48,7 @@ from .tui import ( Separator, TuiSession, ) -from .tui.emoji_width import calibrate as calibrate_emoji_width +from .tui.emoji_width import install as install_emoji_widths from .tui.spinner import spinner_interval from .widgets import battery_cell @@ -379,13 +379,13 @@ async def run_menu(ctx: AppContext) -> None: Args: ctx: The shared application context. """ - # Measure how this terminal renders emoji and align Rich to it, before - # prompt_toolkit takes over the screen. This keeps every panel border — and - # every chat bubble — aligned regardless of the terminal's emoji widths. Skipped - # outright on a platform that never draws emoji (PicoCalc): there is nothing to - # calibrate, and the probe writes escape sequences the console font can't shape. + # Reserve two cells for every glyph that may be drawn as an emoji, in every width + # authority, before prompt_toolkit lays out its first frame: the session pins each such + # glyph into its reservation, so rows line up whatever the terminal's font does with it. + # Skipped outright on a platform that never draws emoji (PicoCalc), where every glyph is + # one its own font has verified and the stock widths are already exact. if get_platform().emoji: - calibrate_emoji_width() + install_emoji_widths() header_cache: dict = {} session = TuiSession(header=lambda cols: _header(ctx, header_cache, cols)) diff --git a/meshterm/ui/tui/colsnap.py b/meshterm/ui/tui/colsnap.py index 59f4b32..9217281 100644 --- a/meshterm/ui/tui/colsnap.py +++ b/meshterm/ui/tui/colsnap.py @@ -1,10 +1,9 @@ # SPDX-License-Identifier: Apache-2.0 """Pin a written row's columns to the terminal's own grid, so no glyph can shift it. -Every other correction in this package answers *how many cells does this glyph take* — -:mod:`~meshterm.ui.tui.emoji_width` is that question's whole home, two width authorities and -two hand-curated exception sets deep. This module answers the one that actually decides -whether a row lines up: **which column does the next glyph land in?** +:mod:`~meshterm.ui.tui.emoji_width` answers *how many cells does this glyph take*. This +module answers the question that actually decides whether a row lines up: **which column does +the next glyph land in?** A row reaches the terminal as one run of characters, and the terminal advances its own cursor by its own idea of each glyph's width. Where that idea differs from ours by a cell, @@ -18,13 +17,12 @@ Whether a font draws a given emoji in one cell or two is the font's business rat codepoint's; there is no table that is right on every terminal; and no escape sequence reports a renderer's width back to the program, because on a split terminal the component that tracks the cursor and the component that paints the glyph are different programs that disagree with -each other (a cursor probe reads the PTY, not the painter — :mod:`~meshterm.ui.tui.emoji_width` -carries that reasoning in full, along with the two curated sets it leaves us with). Curating -exceptions one confirmed glyph at a time lines up the rows somebody has already looked at, and -leaves every name that arrives off the air — written by a stranger, carrying whatever they felt -like typing — exactly as broken as it was. A contact list is the worst case in the app for -precisely that reason: its content is the one thing on screen that nobody can put on a list in -advance. +each other (a cursor probe reads the PTY, not the painter). Curating exceptions one confirmed +glyph at a time — which is what the app used to do — lines up the rows somebody has already +looked at, and leaves every name that arrives off the air, written by a stranger and carrying +whatever they felt like typing, exactly as broken as it was. A contact list is the worst case +in the app for precisely that reason: its content is the one thing on screen that nobody can +put on a list in advance. So this module stops asking the terminal to agree. After any glyph whose drawn width is not certain, the row carries an **absolute column address** — ``CSI n G``, naming the one-based @@ -40,9 +38,10 @@ move anything but itself: * Drawn **wider**, its overhang is written over by whatever the next column holds. Both are one cell of cosmetic damage inside the glyph's own lane, where a shift was a whole row -of it. The curated width sets keep their job — they are what makes the *reservation* right, so -a listed glyph gets neither the gap nor the overwrite — but they stop being what holds the row -together. A glyph nobody has ever classified now costs a cell instead of a row. +of it, and the second never happens in practice: :mod:`~meshterm.ui.tui.emoji_width` reserves +every glyph that may be drawn as an emoji two cells, the most a single glyph draws. Pinning holds +the lanes after a glyph wherever the terminal puts it; the reservation is what makes the text +beside it line up from row to row as well, since a pin can only land where the glyph was measured. **What is certain.** Pinning after a glyph that did not need it is harmless (it addresses the column the cursor is already in) but it is not free: five bytes on the wire and a cluster walk @@ -55,8 +54,8 @@ screen; the walk is paid only where an emoji actually is. Getting an entry in that list wrong costs exactly what today costs anyway — one glyph's row — so the list is a performance claim and never the thing correctness rests on. That is the whole -point of the inversion: the old exception sets had to be *complete* to keep the app aligned, -and this one only has to be *cheap*. +point of the inversion: the exception sets this replaced had to be *complete* to keep the app +aligned, and this one only has to be *cheap*. **Two writers, two spellings of the same pin.** A frame reaches the terminal one of two ways, and each knows something different about where its cursor is: diff --git a/meshterm/ui/tui/emoji_width.py b/meshterm/ui/tui/emoji_width.py index 9333e37..540904f 100644 --- a/meshterm/ui/tui/emoji_width.py +++ b/meshterm/ui/tui/emoji_width.py @@ -1,142 +1,104 @@ # SPDX-License-Identifier: Apache-2.0 -"""Align Rich's emoji cell-width table with the terminal's *actual* rendering. +"""How wide a glyph is: one rule everywhere, and two cells wherever nobody can know. -Emoji column width is not standardized. Whether a variation-selector-16 sequence -such as ``"☀️"`` (a sun, U+2600 U+FE0F) occupies one cell or two depends on -the terminal emulator and the font it uses. Rich follows Unicode TR51 and counts -such a sequence as **2** columns; many terminals — notably Windows ``conhost`` and -Windows Terminal with common fonts — draw it as **1**. When the two disagree, Rich -pads the line assuming the wrong width and the panel border that follows the text -lands a column off (the classic "the sun pushes the right border left" bug). +Emoji column width is not standardized, and it cannot be discovered. Whether a terminal draws a +given emoji in one cell or two is decided by its font, glyph by glyph, with no rule behind the +split: the same terminal draws ``👋`` in one cell and ``📡`` in two, ``☀️`` in one and ``🛩️`` in +two. No escape sequence reports a renderer's width back to the program, and a cursor probe does +not help: on a split terminal (VS Code, Windows Terminal) the component that tracks the cursor and +the component that paints the glyph are different programs that disagree with each other, and a +probe can only ask the first. -There is no static table that is correct on every terminal, and — importantly — -no fully reliable way to *measure* it either. A cursor-position probe only reveals -how the layer that **tracks the cursor** counts the emoji, which on modern split -terminals is a different component than the one that **paints** it: +This module used to answer that with a probe and two hand-curated exception sets — glyphs +confirmed, one at a time, to be drawn narrower or wider than the tables said. That lined up every +row somebody had already looked at and none of the rows nobody had, and a contact's name arrives +off the air, written by a stranger, carrying whatever they typed. The sets were retired for that +reason, along with the probe that decided when to apply them. -* **VS Code integrated terminal** — ConPTY advances the cursor by 2 for ``☀️``, - but the xterm.js renderer paints it in 1 cell. The border you see follows the - renderer; a probe can only see ConPTY. They disagree, so the probe is wrong here. -* **Windows Terminal** has the same PTY/renderer split. -* A classic **conhost** window or a traditional **POSIX terminal** does its own - cursor tracking and painting, so there a probe *is* the ground truth. +**What replaced them is two halves, and neither needs to know what the font does.** -Because no escape sequence reports the renderer's width back to the program, we -decide the width for :data:`_PROBE` in layers, most trustworthy first: +* **Placement** belongs to :mod:`~meshterm.ui.tui.colsnap`: every glyph whose drawn width is + uncertain is written with its column pinned, so whatever width the terminal gives it, the + glyphs after it land in the columns the app measured. +* **Measurement** belongs here: every glyph that *may* be drawn as an emoji is reserved **two + cells**, the most a single glyph ever draws. Pinned into two cells, a glyph the font draws in two + fills them exactly, and one it draws in one leaves a blank beside it — never an overwrite, never + a shift. That is what lines up the text *after* an emoji from row to row, which pinning alone + cannot do: it pins to the measured column, and the measurement was the thing that varied. -1. An explicit override via ``MESHTERM_EMOJI_VS16=1|2`` — always wins. -2. A fingerprint of renderers known to paint VS16 emoji narrow while their PTY - counts them wide (VS Code, Windows Terminal) — where a probe would mislead. -3. Otherwise, a live cursor probe (``GetConsoleScreenBufferInfo`` on Windows, a - ``ESC[6n`` Device Status Report on POSIX), valid on terminals that track and - paint in the same layer. +**"May be drawn as an emoji"** is read off Unicode, never off a font: -If the chosen width is 1, we patch Rich's string-measuring function so every panel, -table, and text widget — menus and chat bubbles alike — lines up. On non-interactive -runs (tests, piped output, scripted CLI) we do nothing and keep Rich's defaults. +* A glyph built from several codepoints: a variation-selector sequence (``☀️``), a ZWJ sequence + (``👨‍👩‍👧``), a keycap (``1️⃣``), a flag's pair of Regional Indicators (``🇨🇦``). +* A lone codepoint Unicode marks as an emoji with *text* presentation by default (``☀``, ``✈``, + ``❤``) — the set a variation selector promotes to two, which Rich ships as ``narrow_to_wide`` — + and a lone Regional Indicator. A font draws either kind either way. +* An emoji whose default presentation is already emoji (``📡``, ``👋``) measures two in both + width authorities as it stands, and is left to them. -The same "width 1" verdict also narrows a small, curated set of *lone* emoji — single -codepoints like ``👋`` that carry no variation selector yet are still drawn in one cell -here. Unlike a VS16 sequence there is no structural tell (a trailing ``U+FE0F``) and no -rule separating them from the lone emoji this terminal draws two wide (``📡``, ``💬``): -the split is the font's, not the codepoint's, and a cursor probe can't see it. So the set -is an explicit allowlist (:data:`_DEFAULT_NARROW_LONE`, extended without a code change via -``MESHTERM_NARROW_EMOJI``), and — because it is the panel border *prompt_toolkit* places, -not Rich — narrowing a lone emoji patches *both* width authorities, where the VS16 fix only -needed Rich (prompt_toolkit never promoted a VS16 sequence in the first place). +**Two exceptions, both measured at their one cell.** Rich's own single-cell ranges (``©``, ``▶``, +box drawing) are measured by a fast path that never consults a patch, so everything else has to +agree with it. And :data:`_APP_TEXT_MARKS` is the handful of text-default emoji MeshTerm draws +*itself*, as one-cell marks in its own lexicon (``⚠``, ``↕``, ``🗑``). Those are a closed +vocabulary, not content: every glyph in the list is drawn by the app, the app is what checks them +(``meshterm specimen``), and a test holds every such glyph in the source to the list. -Country **flags** (``🇨🇦``, ``🇺🇸``) are the same "width 1" story but need no allowlist, -because for them there *is* a rule: a flag is a grapheme pair of Regional Indicator Symbols -(see :func:`_is_regional_indicator`), which the terminal — and Rich — draw as one two-cell -glyph, while prompt_toolkit's wcwidth counts each indicator as two and so measures the flag -as *four* (a two-column notch, worse than a lone emoji's one). Since the indicators are -shared across flags, they can only be handled as a whole category, never one country at a -time; narrowing every indicator to one cell makes any flag sum to two in both authorities. +**One rule, every authority.** Rich measures in three places — a whole string, a single character, +and the grapheme spans it crops and wraps by — and prompt_toolkit in a fourth. :func:`install` +patches all four from the same rule (:func:`_glyph_width`) at once, and they must never differ: +Rich's segment splitter walks a string a codepoint at a time until a whole-string width meets the +cut it was asked for, and a character measured one inside a string measured two is a cut that walk +never meets. -Some VS16 sequences run the other way — this terminal draws them **two** cells where the "width 1" -verdict would narrow them — and they are carved back out into a curated *wide* set -(:data:`_DEFAULT_WIDE_BASE`, extended without a code change via ``MESHTERM_WIDE_EMOJI``) counted two -in **both** authorities. The renderer is simply not uniform: the probe ``☀️`` paints in one cell, -but ``🛩️`` (a small airplane, U+1F6E9 U+FE0F) paints in two, and narrowing it along with the rest -pulls its chat-row border a column short and smears the line. It is the mirror image of the -lone-narrow allowlist, curated for the same reason: which glyph a terminal draws wide is the font's -business, not the codepoint's, and a cursor probe reads the PTY, not the renderer. Keyed on the -*base* codepoint (the trailing ``U+FE0F`` is skipped regardless). +**Delivery.** prompt_toolkit lays out one codepoint at a time, and folds a zero-width codepoint into +the cell before it while keeping that cell's place, so a mark measured one and followed by a +selector would become a two-cell glyph in a one-cell slot. :class:`ClusterTextControl` hands every +multi-codepoint glyph to it as a single fragment instead, laid out and written as the one glyph it +is. -An emoji **ZWJ sequence** (``🤷‍♂️``, ``👨‍👩‍👧``, ``🏳️‍🌈``) is a different failure and -needs no curation at all, because it has a structural tell of its own: the zero-width joiner says -the codepoints around it are *one glyph*, and a terminal draws it in the width of its base. Rich's -stock measurement already knows this; prompt_toolkit's does not — it sums the parts, so it reserves -three cells for the shrug and **six** for the family, and every one of them pulls the row's right -border in. Both authorities are corrected the same way here (:func:`_joined_out`), and — because -prompt_toolkit builds its screen one *codepoint* at a time — the sequence is also handed to it as a -single fragment (:class:`ClusterTextControl`) so it becomes one ``Char`` rather than four. The -cluster measures whatever its base measures, so the two curated sets above still reach it: listing a -base in the wide set widens the whole sequence with it. - -Two neighbours of that rule are structural too, and corrected at either width. A **skin tone** -(``👍🏽``, or the toned base of ``👨🏻‍💻``) recolours the glyph before it and takes no cell, -as Rich already measures it; wcwidth gives each modifier two, so prompt_toolkit measures it zero -here and a joined run carries it along. And a **stranded joiner** — a MeshCore name cut at its byte -limit just past the ZWJ of ``🏴‍☠️`` — joins nothing (:func:`_joins`): both authorities used to fold -the next character into the flag regardless, and that character is the next lane's padding. - -A **bare** codepoint the authorities *already agree* measures one is not a candidate for that set, -however wide the glyph looks in a font book: without a variation selector asking for emoji -presentation, an emoji outside Emoji_Presentation (``🛣`` U+1F6E3, ``🕸`` U+1F578) draws as a -one-cell text glyph, which is what both authorities said. Forcing one to two reserves a cell the -terminal never draws and pulls the row's border a column *in* — see :data:`_DEFAULT_WIDE_BASE` for -the case that proved it. - -**What this module is no longer answerable for.** A row's *layout* no longer rests on these -tables being right about a glyph. :mod:`~meshterm.ui.tui.colsnap` writes an absolute column -address after anything whose drawn width is uncertain, so a mismeasurement costs one cell inside -that glyph's own lane instead of every lane after it — which is the only answer that can reach a -name off the air, since the curated sets can only ever hold glyphs somebody has already looked -at. What those sets still decide is the *reservation*: a listed glyph gets neither a blank cell -beside it nor an overhang written over, so adding a confirmed one is still worth doing. +**Where it applies.** :func:`install` runs once, when the full-screen app starts on a platform that +draws emoji, before its first frame. Nothing else installs it. Tests, piped output and the command +line keep the stock tables, because nothing pins their output: a glyph reserved two cells and drawn +in one, with no pin after it, pulls the rest of its row a column in. Two jobs beside the measuring belong here, because they are the same question asked of the same -table. **Where a string may be cut** (:func:`cut_cells`, over :func:`clusters`): a lane that +glyphs. **Where a string may be cut** (:func:`cut_cells`, over :func:`clusters`): a lane that truncates a name a codepoint at a time cuts through the middle of a glyph, and the pieces that -leaves — a stranded joiner, half a flag — are exactly what every correction above is written -against. And **what may be drawn at all** (:func:`drawable`): an advert name is written by a -stranger and can carry a newline or an escape, which no width table can make safe. +leaves — a stranded joiner, half a flag — are nothing a terminal draws as measured. And **what may +be drawn at all** (:func:`drawable`): an advert name is written by a stranger and can carry a +newline or an escape, which no width can make safe. """ from __future__ import annotations -import os -import sys +import re import unicodedata from collections.abc import Callable, Iterator +from functools import lru_cache from prompt_toolkit.layout.controls import FormattedTextControl from rich.cells import cell_len -from ...core import win32dll - #: The zero-width joiner: the tell that the codepoints it sits between are **one glyph** #: (``🤷‍♂️``, ``👨‍👩‍👧``), which a terminal draws in the width of the sequence's base. -_ZWJ = "‍" +_ZWJ = "\u200d" #: Variation selector 16, the request for emoji presentation. -_VS16 = "️" +_VS16 = "\ufe0f" + +#: The combining enclosing keycap, which turns the digit or sign before it (and the selector +#: between them) into one keycap glyph: ``1️⃣``. +_KEYCAP = "\u20e3" #: The five Fitzpatrick skin-tone modifiers (U+1F3FB–U+1F3FF). One following an emoji base #: *recolours* that glyph rather than drawing one of its own — ``👍🏽`` is a thumb, not a thumb -#: beside a swatch — so it occupies no cell. Rich's table already says zero; wcwidth says two, -#: which is how ``👨🏻‍💻`` came to cost prompt_toolkit four cells for a two-cell glyph. +#: beside a swatch — so it occupies no cell. Rich's table says zero; wcwidth says two. _MODIFIERS = frozenset(chr(cp) for cp in range(0x1F3FB, 0x1F400)) -#: What may trail a base inside one glyph with no joiner in front of it: its variation selector -#: and its skin tone. A joined run gathers these after its base and after every component. -_EXTENDERS = frozenset((_VS16, *_MODIFIERS)) - -#: Codepoints that never occupy a cell on their own: zero-width joiner and the -#: emoji-presentation variation selector. Skipping them means a base glyph is -#: measured at its own East-Asian width, with no VS16 "promote to 2" step. -_ZERO_WIDTH = (_ZWJ, _VS16) +#: What may trail a base inside one glyph with no joiner in front of it: its variation selector, +#: its skin tone, and a keycap's enclosing mark. A joined run gathers these after its base and +#: after every component. +_EXTENDERS = frozenset((_VS16, _KEYCAP, *_MODIFIERS)) #: Unicode general categories holding nothing a terminal can draw: control characters #: (``Cc``), the format and bidi-control block (``Cf``), surrogates (``Cs``), and the line @@ -144,48 +106,35 @@ _ZERO_WIDTH = (_ZWJ, _VS16) #: space — and keeps the one ``Cf`` codepoint that matters, the zero-width joiner. _UNDRAWABLE = frozenset(("Cc", "Cf", "Cs", "Zl", "Zp")) -#: The probe: a sun. Its base (U+2600) is East-Asian *narrow* (1), and Rich -#: promotes the U+2600+U+FE0F pair to 2. So a terminal that reports 1 for this is -#: exactly the terminal whose borders Rich currently misaligns. -_PROBE = "☀️" - -#: Lone-codepoint emoji this terminal draws in a single cell even though wcwidth and Rich -#: both call them two — the residual of the VS16 story for glyphs that carry no variation -#: selector (``👋`` is the one confirmed here). There is no rule separating these from the -#: emoji the same terminal draws two wide (``📡``, ``💬``, the menu icons): the split is the -#: font's glyph, not the codepoint, and it cannot be probed (a cursor probe reads the PTY, -#: not the renderer — see the module docstring). So this is a curated allowlist, seeded with -#: the confirmed glyph and extended — no code change — through ``MESHTERM_NARROW_EMOJI``. -_DEFAULT_NARROW_LONE = "👋" - -#: Base codepoints this terminal draws *two* cells wide where the width authorities say one — -#: the mirror of :data:`_DEFAULT_NARROW_LONE`, and populated for one reason only: a **VS16 -#: sequence** the narrow-VS16 verdict gets wrong. The renderer is not uniform (see the module -#: docstring): ``☀️``, the calibration probe, paints in one cell, but ``🛩️`` (U+1F6E9 U+FE0F) -#: paints in two, so blanket-narrowing it would pull that row's border a column short. +#: The text-default emoji MeshTerm draws *itself*, as one-cell marks in its own lexicon, and so +#: measures at one cell rather than reserving two. Each is a glyph the app has put on screen +#: deliberately and checks through ``meshterm specimen``, not one that arrived in somebody's +#: name, so its width is a fact about the app's own vocabulary rather than a guess about a font. #: -#: A **bare** codepoint does not belong here, and the attempt to put one here is worth recording -#: because the mistake is inviting. An emoji outside Emoji_Presentation — ``🛣`` U+1F6E3, ``🕸`` -#: U+1F578 — has East-Asian width Neutral, so Rich and wcwidth both measure it one; without a -#: variation selector asking for emoji presentation the font draws it as a one-cell text glyph, -#: which is exactly what they said. Listing ``🛣`` to force it to two made the Trophy case's -#: Longest-distance heading reserve a cell the terminal never drew, pulling that row's right -#: border one column in (JP, 2026-08-09) — while ``🕸``, the same Unicode class one board down and -#: never listed, framed flush the whole time. **Both authorities already agreeing at one is not a -#: bug to correct**; when a row misaligns, check whether the glyph carries U+FE0F before assuming -#: the font overrode the measurement. -#: -#: Which VS16 sequence a terminal paints wide is still the font's business rather than the -#: codepoint's, and a probe reads the PTY and not the renderer, so this stays a curated set, -#: extended without a code change through ``MESHTERM_WIDE_EMOJI``. Keyed on the base codepoint, -#: because the trailing selector is skipped by both authorities either way. -_DEFAULT_WIDE_BASE = "\U0001f6e9" # 🛩 airplane, drawn two-wide with its VS16 selector +#: The list is closed by a test, not by care: every text-default emoji in the package's source +#: must be named here, so a new mark cannot slip into the reserve-two rule unnoticed. A glyph +#: that is not here is reserved two cells, which costs a blank beside it and never a misaligned +#: row — which is why this is a list that may be short, where the sets it replaced had to be +#: complete. +_APP_TEXT_MARKS = frozenset( + ( + "⚠", # ⚠ the warning mark + "↔", # ↔ a two-way link: the path composer, the key legend + "↕", # ↕ reorder + "↩", # ↩ reply, in the chat composer + "⌨", # ⌨ the command line + "⚙", # ⚙ a parameter + "\U0001f5d1", # 🗑 clear, delete + "\U0001f578", # 🕸 a Trophy case record + "\U0001f6e3", # 🛣 a Trophy case record + ) +) -#: Set once :func:`calibrate` has run so repeated calls are cheap no-ops. -_CALIBRATED = False +#: Set once :func:`install` has run, so repeated calls are cheap no-ops. +_INSTALLED = False #: Whether the width authorities have been redirected through this module. Gates -#: :class:`ClusterTextControl`: handing prompt_toolkit a joined sequence as one character is +#: :class:`ClusterTextControl`: handing prompt_toolkit a multi-codepoint glyph as one character is #: only right once something measures that character as one glyph, so the delivery and the #: measurement are switched on together or not at all. _CLUSTERS = False @@ -195,12 +144,9 @@ def _is_regional_indicator(char: str) -> bool: """Whether ``char`` is a single Regional Indicator Symbol — a country flag's building block. The two-letter flags (``🇨🇦``, ``🇺🇸``) are a grapheme pair drawn from this block - (U+1F1E6–U+1F1FF). wcwidth counts each indicator as two, so prompt_toolkit measures a flag - as *four* cells, while Rich and the terminal draw the pair as one two-cell glyph. Narrowing - every indicator to one makes any flag sum to two in both authorities, so a chat row with a - flag frames flush. This is a whole category, not a per-country entry: the indicators are - shared across flags (``🇨🇦`` and ``🇨🇳`` both start with ``C``), so they can only be - handled as a block — listing one flag would half-fix its neighbours and skip the rest. + (U+1F1E6–U+1F1FF). The pair is one glyph; either half alone is a letter-like glyph a font + may draw at either width. It is a whole category, never a per-country entry: the indicators + are shared across flags (``🇨🇦`` and ``🇨🇳`` both start with ``C``). """ return len(char) == 1 and 0x1F1E6 <= ord(char) <= 0x1F1FF @@ -212,49 +158,79 @@ def _joins(char: str) -> bool: all at U+2000 and above — so that is the test. What it rules out is a **stranded** joiner. A MeshCore advert name is cut at a byte limit, and a name that ended on a joined emoji (``That's So Fetch 🏴‍☠️``) arrives cut just past the joiner, followed by whatever the row - puts after the name: a space of lane padding, the next lane. Both width authorities used - to fold that character into the flag regardless, so the padding measured nothing while the - terminal still drew it, and every lane after the name started a column late. + puts after the name: a space of lane padding, the next lane. Folding that character into the + flag would take its cell from it while the terminal still drew it. It also leaves the joiners + of the scripts that use them between letters (the Indic conjuncts, below U+2000) as the + letters they are. """ - return char >= " " and not char.isspace() + return char >= "\u2000" and not char.isspace() -def _stranded_out(text: str) -> str: - """Drop every joiner that joins nothing (see :func:`_joins`), leaving real sequences whole.""" - last = len(text) - 1 - return "".join( +@lru_cache(maxsize=1) +def _reserved() -> frozenset[str]: + """Every lone codepoint that is reserved two cells, read off Unicode once. + + The text-default emoji Rich's own table lists (``narrow_to_wide``, the codepoints a variation + selector promotes), less the ones measured by Rich's single-cell fast path and the app's own + marks (see the module docstring for both), plus the Regional Indicators. + """ + from rich._unicode_data import load + from rich.cells import _SINGLE_CELLS + + text_default = frozenset( char - for index, char in enumerate(text) - if char != _ZWJ or (index < last and _joins(text[index + 1])) + for char in load("auto").narrow_to_wide + if not char.isascii() and char not in _SINGLE_CELLS and char not in _APP_TEXT_MARKS ) + return text_default | frozenset(chr(cp) for cp in range(0x1F1E6, 0x1F200)) -def _joined_out(text: str) -> str: - """Drop every codepoint a ZWJ folds into the glyph before it, leaving the bases behind. +@lru_cache(maxsize=1) +def _probe() -> re.Pattern[str]: + """Matches the first codepoint in a string that could make :func:`_glyph_width` differ. - An emoji ZWJ sequence is one grapheme and the terminal draws it in one glyph's worth of - cells — its base's. ``"🤷‍♂️"`` is a shrug, not a shrug beside a male sign, so measuring it - means measuring ``"🤷"`` and ignoring what the joiner attached. Removing the joined - codepoints (and the joiners) leaves a string the ordinary per-codepoint rules — the wide - set, the narrow allowlist, the flag category — measure correctly, so a cluster inherits - whatever its base is worth and every existing lever still reaches it. - - A stranded joiner (see :func:`_joins`) goes on its own: the character after it was never - part of the glyph, and keeps its cell. Variation selectors and skin tones are left in - place — both authorities give them zero. + A string with none of these measures exactly as the stock tables say, so it is handed to + them untouched — one C-speed scan instead of a cluster walk, which is nearly every string. """ - kept = [] - joined = False # the previous codepoint was a joiner, so this one may belong to that glyph - for char in text: - if char == _ZWJ: - joined = True - continue - if joined: - joined = False - if _joins(char): - continue - kept.append(char) - return "".join(kept) + chars = {_ZWJ, _VS16, _KEYCAP, *_MODIFIERS, *_reserved()} + return re.compile("[" + "".join(re.escape(char) for char in sorted(chars)) + "]") + + +def _override(char: str) -> int | None: + """What this module says a lone codepoint measures, or ``None`` to take the stock answer. + + A skin tone takes no cell (wcwidth gives it two). A reserved codepoint (:func:`_reserved`) + takes two. Everything else is the stock table's business. + """ + if char in _MODIFIERS: + return 0 + if char in _reserved(): + return 2 + return None + + +def _glyph_width(glyph: str, stock: Callable[[str], int]) -> int: + """The cells one glyph is reserved — the single rule every width authority is patched with. + + A glyph built from several codepoints that carries a joiner, a variation selector or a keycap, + or is a flag's indicator pair, is reserved two. Anything else is its base codepoint: a skin tone + rides along with its base, so a toned emoji measures what the emoji does, and a letter wearing a + stray modifier stays a letter. + + Args: + glyph: One glyph, as :func:`clusters` splits it. + stock: The stock width of a single codepoint, in the authority being patched. + + Returns: + The glyph's width in cells. + """ + base = glyph[0] + if len(glyph) > 1 and ( + _ZWJ in glyph or _VS16 in glyph or _KEYCAP in glyph or _is_regional_indicator(base) + ): + return 2 + width = _override(base) + return max(0, stock(base)) if width is None else width def clusters(text: str) -> Iterator[str]: @@ -262,14 +238,15 @@ def clusters(text: str) -> Iterator[str]: The same run :func:`_join_clusters` gathers out of a laid-out line, read off a plain string instead: a base, whatever extends it (:data:`_EXTENDERS` — a variation selector, a - skin tone), then any number of *joiner + component + extenders* groups, plus the - Regional Indicator **pair** a country flag is drawn from. A joiner that joins nothing - (:func:`_joins`) is a cluster of its own, because it is not part of the glyph before it. + skin tone, a keycap's enclosing mark), then any number of *joiner + component + extenders* + groups, plus the Regional Indicator **pair** a country flag is drawn from. A joiner that + joins nothing (:func:`_joins`) is a cluster of its own, because it is not part of the glyph + before it. - The unit a lane has to cut on: every correction in this module measures a *whole* glyph, - so a cut through the middle of one leaves two pieces neither authority can measure — a - lone Regional Indicator drawn as a letter, a skin tone with no base, or the stranded - joiner that goes on to swallow the ellipsis the cut appended. See :func:`cut_cells`. + The unit a lane has to cut on and a width is measured by: a cut through the middle of a + glyph leaves pieces that are nothing a terminal draws as measured — a lone Regional + Indicator drawn as a letter, a skin tone with no base, or a stranded joiner. See + :func:`cut_cells`. """ index = 0 count = len(text) @@ -297,15 +274,13 @@ def cut_cells(text: str, width: int) -> str: and the reason it cannot simply walk the string a character at a time: a MeshCore name is written by whoever owns the radio, so it arrives carrying whatever emoji they typed, and a codepoint-by-codepoint cut lands inside one about as often as not. Cutting - ``"👨‍👩‍👧"`` after its first joiner leaves a stranded joiner at the end of the lane, - which then folds the ellipsis the caller appends into the glyph before it - (:func:`_joined_out`) — the ellipsis measures nothing, the terminal draws it anyway, and - every lane after the name starts a column late. Cutting a flag in half leaves a lone - Regional Indicator, which is a letter rather than half a flag. + ``"👨‍👩‍👧"`` after its first joiner leaves a stranded joiner at the end of the lane, and + cutting a flag in half leaves a lone Regional Indicator, which is a letter rather than half + a flag. So the prefix is built out of whole :func:`clusters`, and a trailing joiner — stranded - by the cut itself, or already stranded in the name — is dropped rather than left to - swallow whatever comes next. + by the cut itself, or already stranded in the name — is dropped rather than left beside + whatever the caller appends. Args: text: The text to cut. @@ -333,9 +308,9 @@ def drawable(text: str) -> str: A name comes off the air from a stranger's radio, and nothing on the way in promises it holds only characters. A newline in one ends the row mid-lane; an ``ESC`` starts an escape sequence in the middle of a screen the app composed; a bidi override (``U+202E``) - reverses everything drawn after it. All three measure zero cells and none of them is a - width bug that any allowlist here could fix — they are text that must not reach the - terminal at all, so a lane folds them to the one character that is always safe. + reverses everything drawn after it. None of them is a width question at all — they are + text that must not reach the terminal, so a lane folds them to the one character that is + always safe. Folded: control characters, the format and bidi-control block, surrogates, and the line and paragraph separators. **Kept:** the zero-width joiner, which is format-class too and @@ -348,396 +323,110 @@ def drawable(text: str) -> str: ) -def _narrow_lone_set() -> frozenset[str]: - """The lone-codepoint emoji to measure as one cell (see :data:`_DEFAULT_NARROW_LONE`). +def install() -> None: + """Patch every width authority with the reserve-two rule, once, before the first frame. - ``MESHTERM_NARROW_EMOJI`` overrides the default outright: set it to the full list of - glyphs you have confirmed this terminal draws narrow (e.g. ``MESHTERM_NARROW_EMOJI=👋🤙``), - or to the empty string to turn lone-emoji narrowing off entirely. Only consulted once the - renderer is judged to draw emoji narrow in the first place (see :func:`calibrate`), so on a - terminal that draws emoji two cells wide the whole allowlist is inert and nothing regresses. + Rich's whole-string width, its single-character width (in both modules that read it) and its + grapheme spans, and prompt_toolkit's width cache, all from :func:`_glyph_width` — see the + module docstring for why all four and why at once. Each patch hands a string with nothing + reservable in it straight to the stock code it replaced, so the common case pays one scan. + + Safe to call more than once; only the first call does anything. Called from the menu loop on + a platform that draws emoji; see the module docstring for why nothing else calls it. """ - return frozenset(os.environ.get("MESHTERM_NARROW_EMOJI", _DEFAULT_NARROW_LONE)) - - -def _wide_base_set() -> frozenset[str]: - """The base codepoints to measure two cells wide (see :data:`_DEFAULT_WIDE_BASE`). - - ``MESHTERM_WIDE_EMOJI`` overrides the default outright: set it to the base glyphs you have - confirmed this terminal draws two wide (e.g. ``MESHTERM_WIDE_EMOJI=🛩🛣``), or to the empty - string to trust the narrow-VS16 verdict for every sequence. A trailing variation selector is - dropped, so pasting the whole rendered glyph (``🛩️``) still keys on the base and never counts - the zero-width selector as a wide cell itself. Only consulted on the width-1 path (see - :func:`calibrate`); a terminal that already draws emoji two wide measures these right on its - own, and nothing here would have narrowed them. - """ - listed = frozenset(os.environ.get("MESHTERM_WIDE_EMOJI", _DEFAULT_WIDE_BASE)) - return listed - frozenset(_ZERO_WIDTH) - - -def calibrate(*, force_width: int | None = None) -> None: - """Measure the terminal's emoji width once and patch Rich to match. - - Safe to call more than once (only the first call does work) and safe to call - when there is no real terminal (it becomes a no-op). Call this after the - terminal is available but *before* a full-screen renderer takes it over. - - Args: - force_width: Skip measuring and act as if the terminal rendered the probe - at this width. For tests; production code omits it. - """ - global _CALIBRATED - if _CALIBRATED: + global _INSTALLED, _CLUSTERS + if _INSTALLED: return - _CALIBRATED = True + _INSTALLED = True - width = force_width if force_width is not None else _decide_vs16_width() - # width 1 -> the renderer draws emoji narrow: stop Rich promoting VS16 sequences and - # narrow the curated lone-codepoint emoji this terminal also draws in one cell. - # width 2 -> Rich's default already matches, so its table is left alone (narrowing here - # would instead pull a correctly-wide emoji's border a column short) — but prompt_toolkit - # still sums a ZWJ sequence's parts and a skin tone's cells at either width, and both - # authorities let a stranded joiner swallow the character after it, which is wrong on - # every terminal, so those rules are installed on their own. - # unknown/None -> no terminal to be aligned with; touch nothing. - if width == 1: - _install_terminal_widths(_narrow_lone_set(), _wide_base_set()) - elif width == 2: - _install_cluster_widths() - - -def _decide_vs16_width() -> int | None: - """Decide how wide the renderer paints :data:`_PROBE`, in order of trust. - - Returns: - ``1`` or ``2`` when we have a confident answer, else ``None`` (leave Rich - untouched). See the module docstring for why a probe alone is not enough. - """ - override = os.environ.get("MESHTERM_EMOJI_VS16") - if override in ("1", "2"): - return int(override) - - # Renderers that paint VS16 emoji in one cell while their PTY counts two, so a - # cursor probe reads the PTY's (wrong) answer. Fingerprint them directly. - if os.environ.get("TERM_PROGRAM") == "vscode" or os.environ.get("WT_SESSION"): - return 1 - - # Everywhere else, the terminal tracks and paints in the same layer: measure it. - return _measure_probe_width() - - -def _measure_probe_width() -> int | None: - """Return the column width the live terminal gives :data:`_PROBE`, or ``None``. - - ``None`` means the width could not be measured (not a terminal, the console - call failed, or the escape reply timed out) — callers treat that as "don't - touch Rich". - """ - if not (sys.stdout and sys.stdout.isatty()): - return None - try: - if os.name == "nt": - return _win_probe_width(_PROBE) - return _posix_probe_width(_PROBE) - except Exception: # noqa: BLE001 - any probe failure must fall back cleanly - return None - - -# --- Windows: read the cursor cell column from the console API ---------------- - - -def _win_probe_width(probe: str) -> int | None: - """Measure ``probe`` width via the Windows console cursor position.""" - import ctypes - from ctypes import wintypes - - class _COORD(ctypes.Structure): - _fields_ = [("X", wintypes.SHORT), ("Y", wintypes.SHORT)] - - class _SMALL_RECT(ctypes.Structure): - _fields_ = [ - ("Left", wintypes.SHORT), - ("Top", wintypes.SHORT), - ("Right", wintypes.SHORT), - ("Bottom", wintypes.SHORT), - ] - - class _CSBI(ctypes.Structure): - _fields_ = [ - ("dwSize", _COORD), - ("dwCursorPosition", _COORD), - ("wAttributes", wintypes.WORD), - ("srWindow", _SMALL_RECT), - ("dwMaximumWindowSize", _COORD), - ] - - # A private handle: declaring these signatures on `ctypes.windll.kernel32` would - # rewrite them for prompt_toolkit too, which calls the same function object with a - # struct class of its own. See meshterm.core.win32dll. - kernel32 = win32dll.kernel32() - # Declare signatures explicitly. Without this ctypes assumes a 32-bit int - # return, which truncates the 64-bit console HANDLE and makes every - # subsequent call fail — silently reducing the probe to "no measurement". - kernel32.GetStdHandle.argtypes = [wintypes.DWORD] - kernel32.GetStdHandle.restype = wintypes.HANDLE - kernel32.GetConsoleScreenBufferInfo.argtypes = [ - wintypes.HANDLE, - ctypes.POINTER(_CSBI), - ] - kernel32.GetConsoleScreenBufferInfo.restype = wintypes.BOOL - - STD_OUTPUT_HANDLE = wintypes.DWORD(-11).value - handle = kernel32.GetStdHandle(STD_OUTPUT_HANDLE) - - def cursor_x() -> int | None: - info = _CSBI() - if not kernel32.GetConsoleScreenBufferInfo(handle, ctypes.byref(info)): - return None - return info.dwCursorPosition.X - - sys.stdout.write("\r") - sys.stdout.flush() - start = cursor_x() - if start is None: - return None - sys.stdout.write(probe) - sys.stdout.flush() - end = cursor_x() - # Erase the probe so it never flashes into the user's scrollback. - sys.stdout.write("\r" + " " * (2 if end is None else max(0, end - start)) + "\r") - sys.stdout.flush() - if end is None: - return None - return end - start - - -# --- POSIX: ask the terminal with a Device Status Report ---------------------- - - -def _posix_probe_width(probe: str, timeout: float = 0.3) -> int | None: - """Measure ``probe`` width via a cursor-position report (``ESC[6n``).""" - import re - import select - import termios - import tty - - fd = sys.stdin.fileno() - old = termios.tcgetattr(fd) - try: - tty.setraw(fd) - # Column 1, print the probe, then ask where the cursor landed. - sys.stdout.write("\r" + probe + "\x1b[6n") - sys.stdout.flush() - reply = "" - while "R" not in reply: - ready, _, _ = select.select([fd], [], [], timeout) - if not ready: - break - reply += os.read(fd, 32).decode(errors="ignore") - finally: - termios.tcsetattr(fd, termios.TCSADRAIN, old) - # Erase the probe from the line. - sys.stdout.write("\r\x1b[K") - sys.stdout.flush() - - match = re.search(r"\x1b\[\d+;(\d+)R", reply) - if not match: - return None - # We started at column 1, so the reported column is 1 + width. - return int(match.group(1)) - 1 - - -# --- the patch ---------------------------------------------------------------- - - -def _make_cell_len(narrow: frozenset[str], wide: frozenset[str]) -> Callable[[str, str], int]: - """Build Rich's ``_cell_len`` replacement: the terminal's true width for every string. - - Corrections fold into the one measuring loop the whole render pipeline reads: - - * **VS16 promotion** — a variation selector (:data:`_ZERO_WIDTH`) is skipped, so a narrow - base (``☀`` = 1) is measured on its own instead of promoted to two by a trailing - ``U+FE0F``. This is the general emoji-presentation fix (see the module docstring). - * **Wide VS16 exceptions** — a base codepoint in ``wide`` (``🛩`` and friends) that this - terminal draws two cells despite the narrow-VS16 verdict is forced back to two, so its - selector-skipped base does not collapse to one and smear the row. - * **Curated lone emoji** — a single codepoint in ``narrow`` (``👋`` and friends) that Rich - would call two but this terminal draws in one is counted as one, so a chat row carrying - it pads to a flush right border instead of a column-short notch. - * **Flag indicators** — a Regional Indicator (:func:`_is_regional_indicator`) counts as one, - so a country flag sums to two (Rich already agrees here; this keeps the two loops in step). - * **ZWJ sequences** — the codepoints a joiner folds in are dropped first - (:func:`_joined_out`), so ``"🤷‍♂️"`` is measured as the one glyph it is. Rich's stock - loop does this and the selector-skipping replacement above would otherwise lose it, - counting the male sign as a cell of its own and pulling the row's border a column in. - A *stranded* joiner (:func:`_joins`) folds in nothing, where the stock loop would have - swallowed the padding space after a byte-cut name. - - Everything else keeps its ``get_character_cell_size`` value, so a lone emoji the terminal - *does* draw two wide (a menu icon like ``📡``, never placed in ``narrow``) is left alone. - """ - import rich.cells as cells - - get_size = cells.get_character_cell_size - - def _cell_len(text: str, unicode_version: str = "auto") -> int: - total = 0 - for char in _joined_out(text) if _ZWJ in text else text: - if char in _ZERO_WIDTH: - continue - if char in wide: - total += 2 - elif char in narrow or _is_regional_indicator(char): - total += 1 - else: - total += get_size(char, unicode_version) - return total - - return _cell_len - - -def _make_pt_cache(narrow: frozenset[str], wide: frozenset[str], *, flags: bool = True) -> object: - """Build a prompt_toolkit char-width cache that measures every ``narrow`` glyph as one. - - prompt_toolkit is the authority that *places the panel's right border*: it lays the - composed frame into its screen buffer stepping a cursor by ``get_cwidth`` (backed by - ``_CHAR_SIZES_CACHE``), so unless it too measures ``👋`` as one, it positions the border a - column past where the terminal draws the narrow glyph — the notch survives even with Rich - corrected. The subclass returns one for a listed lone glyph (or a flag's Regional Indicator) - and defers everything else to the stock ``wcwidth`` logic; because the base ``__missing__`` - sums per character through the cache, a whole chat line ``"Bob 👋 hi"`` inherits the one-cell - ``👋`` for free, and a flag ``"🇨🇦"`` sums its two one-cell indicators to a flush two. - - A **ZWJ sequence** is measured as the single glyph it is, by summing only the bases the - joiner left behind (:func:`_joined_out`) — so ``"🤷‍♂️"`` is two rather than three and - ``"👨‍👩‍👧"`` two rather than six. It runs through this same cache, so a base in ``wide`` or - ``narrow`` still sets the sequence's width. Delivering it is the other half of the job: - prompt_toolkit lays out one codepoint at a time, so the sequence also has to reach it as a - single fragment — see :class:`ClusterTextControl`. - - A ``wide`` base codepoint (``🛩``) is the reverse case and matters most here: prompt_toolkit's - wcwidth already calls a VS16 base one on its own (it never promoted the pair), so without this - the airplane's border lands a column short of where the terminal paints its two-cell glyph. The - subclass forces such a base to two; the same per-character summation then gives ``"🛩️ hi"`` its - correct width, the trailing selector staying zero. - - A **skin tone** (:data:`_MODIFIERS`) measures zero, as Rich already has it: wcwidth calls - each modifier two, so ``👍🏽`` cost four cells here and the joined ``👨🏻‍💻`` four as well. - At zero, prompt_toolkit's own layout folds the modifier into the cell before it, the way it - folds a variation selector — and :class:`ClusterTextControl` carries it inside a joined run. - - Args: - narrow: lone codepoints to measure as one cell. - wide: base codepoints to measure as two. - flags: whether to narrow Regional Indicators so a country flag sums to two. Off on the - width-2 path, where the terminal's own flag rendering has not been confirmed and a - renderer without flag glyphs draws the two indicators as four cells of its own. - """ - import prompt_toolkit.utils as ptu - - base_cache_cls = type(ptu._CHAR_SIZES_CACHE) - - class _NarrowLoneCache(base_cache_cls): # type: ignore[valid-type, misc] - def __missing__(self, string: str) -> int: - if _ZWJ in string: - # One glyph, however many codepoints: measure the bases the joiner left - # behind, through this same cache so every rule above still applies. Only - # a short result is kept — a whole *line* holding a sequence is cheap to - # recompute and would otherwise slip past the base class's long-string - # rotation and grow the cache without bound. - total = sum(self[char] for char in _joined_out(string)) - if len(string) <= self.LONG_STRING_MIN_LEN: - self[string] = total - return total - if string in _MODIFIERS: - self[string] = 0 - return 0 - if string in wide: - self[string] = 2 - return 2 - if string in narrow or (flags and _is_regional_indicator(string)): - self[string] = 1 - return 1 - return super().__missing__(string) - - return _NarrowLoneCache() - - -def _install_terminal_widths(narrow: frozenset[str], wide: frozenset[str]) -> None: - """Redirect both width authorities through the terminal-aligned measurement, in one shot. - - Rich's ``cell_len`` delegates to the module-level ``_cell_len``; replacing that (and - clearing the memoised wrapper) steers every downstream caller — panels, tables, text — - through :func:`_make_cell_len`. prompt_toolkit is patched too, because it is what places the - border (see :func:`_make_pt_cache`): a lone emoji, a wide-VS16 exception, or a flag all need - it, and even empty ``narrow``/``wide`` sets still leave the flag-indicator category to - correct, so the cache is always swapped. Both run at :func:`calibrate` time, before a frame is - drawn, so no stale measurement or ``Char`` is ever painted. - - Measuring a ZWJ sequence as one glyph is half of that job, so this is also what arms the - other half — :class:`ClusterTextControl` only merges once something is measuring what it - merges. - """ - global _CLUSTERS import prompt_toolkit.utils as ptu import rich.cells as cells + import rich.segment as segment from .render import _ANSI_CACHE + probe = _probe() + stock_size = cells.get_character_cell_size + stock_cell_len = cells._cell_len + stock_split = cells.split_graphemes + single_cells = cells._is_single_cell_widths + + @lru_cache(maxsize=4096) + def get_character_cell_size(character: str, unicode_version: str = "auto") -> int: + width = _override(character) + return stock_size(character, unicode_version) if width is None else width + + def _cell_len(text: str, unicode_version: str = "auto") -> int: + if single_cells(text): + return len(text) + if probe.search(text) is None: + return stock_cell_len(text, unicode_version) + return sum( + _glyph_width(glyph, lambda char: stock_size(char, unicode_version)) + for glyph in clusters(text) + ) + + def split_graphemes(text: str, unicode_version: str = "auto") -> tuple[list, int]: + if probe.search(text) is None: + return stock_split(text, unicode_version) + spans: list[tuple[int, int, int]] = [] + total = 0 + index = 0 + for glyph in clusters(text): + end = index + len(glyph) + width = _glyph_width(glyph, lambda char: stock_size(char, unicode_version)) + if width == 0 and spans: + # Zero-width pieces belong to the glyph before them, as Rich's own splitter has it, + # so a crop never lands between a letter and its mark. + start, _end, size = spans[-1] + spans[-1] = (start, end, size) + else: + spans.append((index, end, width)) + total += width + index = end + return spans, total + + class _ReservingCache(type(ptu._CHAR_SIZES_CACHE)): # type: ignore[misc] + """prompt_toolkit's width cache, answering from :func:`_glyph_width`.""" + + def __missing__(self, string: str) -> int: + if len(string) == 1: + width = _override(string) + if width is None: + return super().__missing__(string) + self[string] = width + return width + if probe.search(string) is None: + return super().__missing__(string) + total = sum(_glyph_width(glyph, self.__getitem__) for glyph in clusters(string)) + # A whole line holding an emoji is cheap to recompute, and would otherwise slip past + # the base class's rotation of long strings and grow the cache without bound. + if len(string) <= self.LONG_STRING_MIN_LEN: + self[string] = total + return total + + cells.get_character_cell_size = get_character_cell_size + segment.get_character_cell_size = get_character_cell_size + cells._cell_len = _cell_len + cells.split_graphemes = split_graphemes cells.cached_cell_len.cache_clear() - cells._cell_len = _make_cell_len(narrow, wide) - ptu._CHAR_SIZES_CACHE = _make_pt_cache(narrow, wide) + ptu._CHAR_SIZES_CACHE = _ReservingCache() _CLUSTERS = True # The measurement just changed under every renderable, so anything rasterized before - # calibration (nothing, in the normal boot order — but never trust that) is stale. - _ANSI_CACHE.clear() - - -def _install_cluster_widths() -> None: - """Correct what is wrong at *either* emoji width: joined runs, skin tones, stranded joiners. - - A terminal that draws ``☀️`` in two cells needs none of the narrowing above — Rich's table - already matches what it paints, and so does prompt_toolkit's for everything that stands on - its own. A joined sequence is the exception: prompt_toolkit adds up its codepoints wherever - it runs, so ``👨‍👩‍👧`` reserves six cells for a two-cell glyph on the widest terminal as - surely as on the narrowest, and a skin tone costs it two more on any of them. So - prompt_toolkit's cache is swapped, with both curated sets empty and the flag category left - alone, since neither has been confirmed here. - - Rich's *table* is left as it is — it measures a real cluster and a skin tone right on its - own — and only its loop is wrapped, for the one string it gets wrong: it skips whatever - follows a joiner without asking what that is, so the padding after a byte-cut ``🏴‍`` - measured nothing (see :func:`_joins`). The wrapper takes the stranded joiners out before the - stock loop sees the string; every other string reaches it untouched. - """ - global _CLUSTERS - import prompt_toolkit.utils as ptu - import rich.cells as cells - - from .render import _ANSI_CACHE - - stock = cells._cell_len - - def _cell_len(text: str, unicode_version: str = "auto") -> int: - return stock(_stranded_out(text) if _ZWJ in text else text, unicode_version) - - cells.cached_cell_len.cache_clear() - cells._cell_len = _cell_len - ptu._CHAR_SIZES_CACHE = _make_pt_cache(frozenset(), frozenset(), flags=False) - _CLUSTERS = True + # (nothing, in the normal boot order — but never trust that) is stale. _ANSI_CACHE.clear() class _Cluster(str): - """An emoji ZWJ sequence that must reach the screen buffer as **one** character. + """A multi-codepoint glyph that must reach the screen buffer as **one** character. prompt_toolkit builds its screen a codepoint at a time — ``for c in text`` in - ``Window._copy_body`` — and gives each one its own cell-occupying ``Char``. A joined - sequence measured correctly at two cells is therefore still laid out as a two-cell shrug - followed by a one-cell male sign, and the border lands three columns along from a glyph the - terminal drew in two. Iterating a cluster yields the whole sequence instead of its parts, so - that loop makes a single ``Char`` of it: one cell pair in prompt_toolkit's arithmetic, every - codepoint written to the terminal back to back, and one composed glyph on the screen. + ``Window._copy_body`` — and gives each one its own cell-occupying ``Char``. A glyph measured + correctly at two cells is therefore still laid out as its parts: a shrug followed by a male + sign, a flag as two letters, a warning mark with a selector folded into a slot one cell wide. + Iterating a cluster yields the whole glyph instead, so that loop makes a single ``Char`` of it: + one width in prompt_toolkit's arithmetic, every codepoint written to the terminal back to back, + and one composed glyph on the screen. It is a ``str`` subclass rather than a wrapper because it has to survive as ordinary text through everything else prompt_toolkit does with a fragment — joining, slicing, comparing. @@ -751,32 +440,30 @@ class _Cluster(str): yield str(self) +#: Codepoints whose presence in a laid-out line means some glyph on it is built from several. +_JOINERS = frozenset((_ZWJ, *_EXTENDERS)) + + def _join_clusters(line: list) -> list: - """Merge each multi-glyph emoji in one screen line into a single :class:`_Cluster` fragment. + """Merge each multi-codepoint glyph in one screen line into a single :class:`_Cluster` fragment. The line arrives one codepoint per fragment (that is what :class:`~prompt_toolkit.formatted_text.ANSI` produces), so a glyph built from several is a run - to be gathered, the same run :func:`clusters` reads off a plain string. Two shapes need it: + to be gathered, the same run :func:`clusters` reads off a plain string: a base and whatever + extends it, a flag's indicator pair, and any number of *joiner + component + extenders* groups. - * **A ZWJ sequence**: a base, its optional variation selector or skin tone - (:data:`_EXTENDERS`), then any number of *joiner + component + optional extenders* groups. - prompt_toolkit would otherwise give each component a cell of its own. - * **A flag**: a pair of Regional Indicators. Each indicator takes a cell of its own in - prompt_toolkit's arithmetic, and summing them was already made right; what merging adds is - that the pair is *written* as one. The renderer's writes are what - :class:`~meshterm.ui.tui.colsnap.PinnedOutput` pins, and a pin between the two halves of - a flag is a cursor move between them, after which a terminal no longer joins them into - one glyph at all. + Every such glyph is merged, a lone selector pair and a toned emoji included. prompt_toolkit + would fold those into the cell before them on its own, but it keeps that cell's place while it + does, so a mark measured one cell and followed by a selector would become a two-cell glyph in a + one-cell slot. And the renderer's writes are what :class:`~meshterm.ui.tui.colsnap.PinnedOutput` + pins: a pin between the two halves of a flag is a cursor move between them, after which a + terminal no longer joins them into one glyph at all. - A run with neither is handed back untouched — including a lone VS16 pair or toned emoji, - which prompt_toolkit already folds into the preceding cell on its own, and a joiner with no - pictograph after it (see :func:`_joins`), which is not a sequence at all. - - The whole line is handed back unchanged when it holds no joiner and no indicator, which is - almost every line; the scan is one pass per line and runs only when a control's content - actually changes. + A joiner with no pictograph after it (see :func:`_joins`) is not part of any glyph, and keeps + its fragment. The whole line is handed back unchanged when nothing on it could start or extend + such a glyph, which is almost every line; the scan runs only when a control's content changes. """ - if not any(item[1] == _ZWJ or _is_regional_indicator(item[1]) for item in line): + if not any(item[1] in _JOINERS or _is_regional_indicator(item[1]) for item in line): return line merged: list = [] index = 0 @@ -785,47 +472,43 @@ def _join_clusters(line: list) -> list: end = index + 1 while end < count and line[end][1] in _EXTENDERS: end += 1 - joined = False if ( _is_regional_indicator(line[index][1]) and end < count and _is_regional_indicator(line[end][1]) ): - joined = True end += 1 while end < count and line[end][1] in _EXTENDERS: end += 1 while end + 1 < count and line[end][1] == _ZWJ and _joins(line[end + 1][1]): - joined = True end += 2 # the joiner and the codepoint it joins while end < count and line[end][1] in _EXTENDERS: end += 1 - if joined: + if end - index > 1: text = "".join(item[1] for item in line[index:end]) merged.append((line[index][0], _Cluster(text))) - index = end else: merged.append(line[index]) - index += 1 + index = end return merged class ClusterTextControl(FormattedTextControl): - """A :class:`~prompt_toolkit.layout.controls.FormattedTextControl` that keeps emoji whole. + """A :class:`~prompt_toolkit.layout.controls.FormattedTextControl` that keeps glyphs whole. The control is the last place a screen's text is still arranged in lines and still ours to - touch, which is exactly what merging a sequence needs: ``split_lines`` runs before this - point and returns plain ``str`` parts, so a :class:`_Cluster` marked any earlier would not - survive to be laid out. Everything downstream — the width lookup, the wrapping, the screen - buffer — reads the merged lines. + touch, which is exactly what merging a glyph needs: ``split_lines`` runs before this point and + returns plain ``str`` parts, so a :class:`_Cluster` marked any earlier would not survive to be + laid out. Everything downstream — the width lookup, the wrapping, the screen buffer — reads the + merged lines. - Merging is gated on :func:`calibrate` having run, because handing prompt_toolkit a sequence - as one character is only right once its cache measures that character as one glyph. Off that - path (a test, a piped run, a terminal we could not measure) this is its base class exactly. + Merging is gated on :func:`install` having run, because handing prompt_toolkit a glyph as one + character is only right once its cache measures that character as one glyph. Off that path + (a test, a platform that draws no emoji) this is its base class exactly. """ def create_content(self, width: int, height: int | None): # type: ignore[override] - """The base class's content, with each line's ZWJ sequences joined into one glyph. + """The base class's content, with each line's multi-codepoint glyphs joined into one. The joining is wrapped around the returned object's line lookup and memoised behind it, because prompt_toolkit hands back the same content object paint after @@ -835,8 +518,8 @@ class ClusterTextControl(FormattedTextControl): # The base class caches its ``UIContent`` per (fragments, width, cursor), so the same # object comes back paint after paint: wrap its line lookup once, and memoise the merge # behind it, rather than re-walking every line of every frame. - if _CLUSTERS and not getattr(content, "_zwj_merged", False): - content._zwj_merged = True # type: ignore[attr-defined] + if _CLUSTERS and not getattr(content, "_clusters_merged", False): + content._clusters_merged = True # type: ignore[attr-defined] source = content.get_line cache: dict[int, list] = {} @@ -850,4 +533,4 @@ class ClusterTextControl(FormattedTextControl): return content -__all__ = ["ClusterTextControl", "calibrate", "clusters", "cut_cells", "drawable"] +__all__ = ["ClusterTextControl", "clusters", "cut_cells", "drawable", "install"] diff --git a/tests/test_emoji_width.py b/tests/test_emoji_width.py index 9b1a30a..e600390 100644 --- a/tests/test_emoji_width.py +++ b/tests/test_emoji_width.py @@ -1,330 +1,322 @@ # SPDX-License-Identifier: Apache-2.0 -"""Terminal-aligned emoji width: VS16 sequences and curated lone-codepoint emoji. +"""Reserve-two emoji widths: one rule in every width authority, and every glyph kept whole. -The border-alignment story lives in :mod:`meshterm.ui.tui.emoji_width`. These tests pin the -two things that make a chat row with ``👋`` frame flush: Rich must *measure* the glyph as one -cell (so it pads the row to the right width) and prompt_toolkit must *count* it as one (so it -places the panel's right border where the terminal actually draws it). An emoji the terminal -draws two wide — a menu icon like ``📡``, never on the allowlist — must stay two in both. +The measuring story lives in :mod:`meshterm.ui.tui.emoji_width`, and the placement half in +:mod:`meshterm.ui.tui.colsnap`. What these tests pin: -The factory helpers (``_make_cell_len``, ``_make_pt_cache``) are exercised in isolation so no -process-global width table is touched; the two :func:`calibrate` tests that do patch the real -tables snapshot and restore them in a ``finally`` so nothing leaks into a later test. +* every glyph that may be drawn as an emoji is reserved two cells, in Rich and in prompt_toolkit; +* the app's own marks, Rich's single-cell ranges and ordinary text keep their stock widths; +* Rich's three measurements and prompt_toolkit's agree about every string, which is what keeps + Rich's segment splitter from walking past a cut it can never meet; +* the app's source draws no text-default emoji its vocabulary does not name; +* a glyph built from several codepoints is split, cut and laid out as the one glyph it is. + +:func:`~meshterm.ui.tui.emoji_width.install` patches process-global tables, so every test that +needs it goes through the ``installed`` fixture, which puts every table and cache back afterwards. """ from __future__ import annotations +import ast +import pathlib +import threading +from collections.abc import Iterator + import prompt_toolkit.utils as ptu +import pytest import rich.cells as cells +import rich.segment as segment +from prompt_toolkit.utils import get_cwidth from meshterm.ui.tui import emoji_width as ew -# One confirmed narrow glyph, plus one the same terminal draws two wide, used throughout. -_WAVE = "👋" # the confirmed lone-codepoint emoji this terminal draws in one cell -_DISH = "📡" # a menu icon the terminal draws two wide — must never be narrowed -_CA = "🇨🇦" # a flag: two Regional Indicators; prompt_toolkit miscounts it as four cells -_CN = "🇨🇳" # a second flag sharing the "C" indicator — the whole category must be handled -_PLANE = "🛩️" # a VS16 sequence this terminal draws two wide despite the narrow-VS16 verdict -_PLANE_BASE = "\U0001f6e9" # its base codepoint (no selector) — what the wide set is keyed on -_ROAD = "\U0001f6e3" # 🛣 a bare text-default emoji: one cell, and both authorities agree -_WEB = "\U0001f578" # 🕸 its untouched sibling one Trophy board down — the control -_SHRUG = "\U0001f937‍♂️" # 🤷‍♂️ a ZWJ sequence: base, joiner, male sign, selector — one glyph -_FAMILY = "\U0001f468‍\U0001f469‍\U0001f467" # 👨‍👩‍👧 three joined people, still one glyph -_ZWJ = "‍" # the joiner itself: the tell that its neighbours are a single glyph -_THUMB = "\U0001f44d\U0001f3fd" # 👍🏽 a base and its skin tone, no joiner — still one glyph -_TECHIE = "\U0001f468\U0001f3fb‍\U0001f4bb" # 👨🏻‍💻 a toned base, then a joined laptop -_FLAG = "\U0001f3f4" # 🏴 a black flag, the base of the pirate flag below -_STRANDED = f"{_FLAG}‍" # 🏴‍☠️ cut after its joiner by a name's byte limit: joins nothing +_ZWJ = "\u200d" # the joiner: the tell that its neighbours are a single glyph +_VS16 = "\ufe0f" # the request for emoji presentation +_WAVE = "\U0001f44b" # 👋 emoji presentation by default: two cells by the stock tables already +_DISH = "\U0001f4e1" # 📡 a menu icon, the same +_CA = "\U0001f1e8\U0001f1e6" # 🇨🇦 a flag: two Regional Indicators, one glyph +_CN = "\U0001f1e8\U0001f1f3" # 🇨🇳 a second flag sharing the "C" indicator +_SUN = "☀" # ☀ a text-default emoji, alone +_PLANE = "\U0001f6e9" # 🛩 another, which a font draws either way +_HEART = "❤" # ❤ another +_ROAD = "\U0001f6e3" # 🛣 text-default too, but one of the app's own marks +_WEB = "\U0001f578" # 🕸 the same +_BIN = "\U0001f5d1" # 🗑 the same +_WARN = "⚠" # ⚠ the same +_SHRUG = f"\U0001f937{_ZWJ}♂{_VS16}" # the shrug: base, joiner, sign, selector +_FAMILY = f"\U0001f468{_ZWJ}\U0001f469{_ZWJ}\U0001f467" # the family: three joined people +_THUMB = "\U0001f44d\U0001f3fd" # a thumb and its skin tone, no joiner: still one glyph +_TECHIE = f"\U0001f468\U0001f3fb{_ZWJ}\U0001f4bb" # the technologist: toned, then joined +_KEYCAP = f"1{_VS16}\u20e3" # a keycap: a digit, a selector and the enclosing mark +_FLAG = "\U0001f3f4" # a black flag, the base of a pirate flag +_STRANDED = f"{_FLAG}{_ZWJ}" # a pirate flag cut after its joiner by a name's byte limit + +#: Strings every authority has to agree about: emoji of every shape, the app's own marks, and the +#: scripts and marks that must come through the rule untouched. +_SAMPLES = ( + "plain ascii", + f"Bob {_FAMILY} x", + f"{_CA}{_CN}", + f"{_SUN}{_VS16}{_SHRUG}", + f"{_SUN} bare, {_PLANE} bare, {_HEART} bare", + f"a{_ZWJ}b", + f"{_STRANDED} x", + f"{_KEYCAP}!", + f"│ {_WARN}{_VS16} warn │ {_WARN} mark │", + f"Tech {_TECHIE} x {_THUMB}{_THUMB}", + "漢字 names", + "ष\u094d\u200dक", # a Devanagari conjunct: a joiner between letters, not pictographs + f"{_CA[0]} a lone indicator", + "e\u0301 a combining accent", + f"{_ROAD}{_WEB}{_BIN} ↔ ↕", +) -def test_narrow_lone_set_defaults_extends_and_disables(monkeypatch) -> None: - """The allowlist seeds to the confirmed glyph; the env var overrides it outright.""" - monkeypatch.delenv("MESHTERM_NARROW_EMOJI", raising=False) - assert _WAVE in ew._narrow_lone_set() +@pytest.fixture +def installed() -> Iterator[None]: + """Install the reserve-two rule for one test, and put every table and cache back after.""" + from meshterm.ui.tui import colsnap + from meshterm.ui.tui.render import _ANSI_CACHE - monkeypatch.setenv("MESHTERM_NARROW_EMOJI", "👋🤙") - assert {"👋", "🤙"} <= ew._narrow_lone_set() - - # Empty string is the off switch — a way back to Rich/pt defaults with no code change. - monkeypatch.setenv("MESHTERM_NARROW_EMOJI", "") - assert ew._narrow_lone_set() == frozenset() - - -def test_wide_base_set_defaults_extends_and_disables(monkeypatch) -> None: - """The wide set seeds to the confirmed bases; the env var overrides it outright.""" - monkeypatch.delenv("MESHTERM_WIDE_EMOJI", raising=False) - # One source only: a VS16 sequence the narrow verdict gets wrong. A *bare* codepoint both - # authorities already measure at one is not a candidate — see test_bare_text_default_emoji. - assert _PLANE_BASE in ew._wide_base_set() - assert not {_ROAD, _WEB} & ew._wide_base_set() - - # ✈ (U+2708) is a second VS16 airplane base; both list cleanly. - monkeypatch.setenv("MESHTERM_WIDE_EMOJI", "\U0001f6e9✈") - assert {"\U0001f6e9", "✈"} <= ew._wide_base_set() - - # Pasting the whole rendered glyph keeps the base but strips the zero-width selector, so - # the selector is never itself counted as a wide cell. - monkeypatch.setenv("MESHTERM_WIDE_EMOJI", _PLANE) - wide = ew._wide_base_set() - assert _PLANE_BASE in wide and "️" not in wide - - # Empty string trusts the narrow-VS16 verdict for every sequence. - monkeypatch.setenv("MESHTERM_WIDE_EMOJI", "") - assert ew._wide_base_set() == frozenset() - - -def test_rich_cell_len_narrows_only_allowlisted_lone_emoji() -> None: - """Rich's replacement measures a listed lone emoji as one, an unlisted one still as two.""" - cell_len = ew._make_cell_len(frozenset(_WAVE), frozenset(_PLANE_BASE)) - - assert cell_len(_WAVE) == 1 - assert cell_len(_DISH) == 2 # unlisted: the terminal draws it wide, so leave it wide - assert cell_len("Bob 👋 hi") == 8 # the narrowed glyph flows through a whole line - # The pre-existing VS16 handling still applies: a narrow base is measured alone, its - # variation selector skipped, so "☀️" stays one cell rather than being promoted to two. - assert cell_len("☀️") == 1 - # A wide-VS16 exception is forced back to two, selector still skipped, so the airplane - # frames flush instead of collapsing to one and smearing the row. - assert cell_len(_PLANE) == 2 - assert cell_len("hi 🛩️") == len("hi ") + 2 - # A ZWJ sequence is one glyph, measured as its base: the codepoints the joiner folds in - # cost nothing, however many of them there are. Rich's stock loop already does this, and - # the selector-skipping replacement must not lose it — counting the male sign as a cell - # of its own is what pulls the row's right border a column in. - assert cell_len(_SHRUG) == 2 - assert cell_len(_FAMILY) == 2 - assert cell_len(f"Bob {_FAMILY} hi") == len("Bob hi") + 2 - - -def test_pt_cache_narrows_only_allowlisted_lone_emoji() -> None: - """prompt_toolkit's cache — the authority that places the border — matches Rich.""" - cache = ew._make_pt_cache(frozenset(_WAVE), frozenset(_PLANE_BASE)) - - assert cache[_WAVE] == 1 - assert cache[_DISH] == 2 - # The base cache sums a multi-char string per character through the cache, so the - # one-cell wave is inherited by any line that contains it. - assert cache["Bob 👋"] == 5 - # The wide-VS16 airplane is the reverse: its base is forced to two, the selector stays - # zero, so the whole glyph (and any line holding it) keeps the terminal's two cells. - assert cache[_PLANE] == 2 - assert cache["🛩️ hi"] == 2 + len(" hi") - # The sequences prompt_toolkit added up part by part: three cells for the shrug and six - # for the family, each of them one two-cell glyph, each pulling a border in by the excess. - assert cache[_SHRUG] == 2 - assert cache[_FAMILY] == 2 - assert cache[f"Bob {_FAMILY} hi"] == len("Bob hi") + 2 - # A lone joiner is still zero, and a codepoint a sequence joins keeps its own width when - # it stands alone — ``↕`` is the reorder icon, not part of anything. - assert cache[_ZWJ] == 0 - assert cache["↕"] == 1 - - -def test_bare_text_default_emoji_are_left_exactly_as_measured() -> None: - """An emoji outside Emoji_Presentation, drawn *bare*, is one cell — and stays one. - - ``🛣`` and ``🕸`` have East-Asian width Neutral, so Rich and wcwidth both measure them at - one; with no variation selector asking for emoji presentation, the font draws a one-cell - text glyph, which is exactly what they said. Both authorities already agreeing is not a - bug to correct: listing ``🛣`` in the wide set made the Trophy case's Longest-distance - heading reserve a cell the terminal never drew and pulled that row's border a column in, - while ``🕸`` — same class, one board down, never listed — framed flush throughout. - """ - cell_len = ew._make_cell_len(frozenset(_WAVE), ew._wide_base_set()) - cache = ew._make_pt_cache(frozenset(_WAVE), ew._wide_base_set()) - for bare in (_ROAD, _WEB): - assert cell_len(bare) == 1 and cache[bare] == 1 - # A heading built around one costs its own cells and no more, so the row frames flush. - caption = cell_len(f"── {_ROAD} Longest distance ──") - assert caption == len("── ") + 1 + len(" Longest distance ──") - # The Trophy case's *visible* gap after the road is not this module's business and must - # not become it: the heading pads the mark out to the widest of the seven disciplines - # (records_screen.discipline_label), which costs a real cell the terminal advances over - # — where widening the glyph here would claim one it does not. - - -def test_flags_measure_two_cells_in_both_authorities() -> None: - """A country flag measures two cells in both width authorities. - - A flag is a Regional Indicator pair: prompt_toolkit's wcwidth calls it four cells, - while Rich and the terminal draw it as one two-cell glyph. Narrowing the - indicators as a category makes every flag sum to two in both — no per-country - allowlist entry, and flags sharing an indicator (🇨🇦 / 🇨🇳) are fixed at once. - """ - # Only the lone-emoji allowlist is passed; flags are handled by category, not by listing. - cell_len = ew._make_cell_len(frozenset(_WAVE), frozenset()) - cache = ew._make_pt_cache(frozenset(_WAVE), frozenset()) - - for flag in (_CA, _CN): - assert cell_len(flag) == 2 - assert cache[flag] == 2 # was 4 unpatched: two indicators at wcwidth 2 each - - # A flag rides a chat line without dragging the border: "eh? 🇨🇦" measures its plain - # cells plus the flag's two. - assert cell_len("eh 🇨🇦") == len("eh ") + 2 - - -def _snapshot() -> tuple: - """Capture the mutable width state :func:`calibrate` patches, to restore afterwards.""" - return (cells._cell_len, ptu._CHAR_SIZES_CACHE, ew._CALIBRATED, ew._CLUSTERS) - - -def _restore(snap: tuple) -> None: - """Put the width authorities (and the once-only flag) back, clearing Rich's memo cache.""" - cells._cell_len, ptu._CHAR_SIZES_CACHE, ew._CALIBRATED, ew._CLUSTERS = snap - cells.cached_cell_len.cache_clear() - - -def test_calibrate_width1_narrows_the_wave_in_both_authorities(monkeypatch) -> None: - """A renderer that draws emoji narrow gets both Rich and pt aligned to the terminal.""" - monkeypatch.delenv("MESHTERM_NARROW_EMOJI", raising=False) - monkeypatch.delenv("MESHTERM_WIDE_EMOJI", raising=False) - from prompt_toolkit.utils import get_cwidth - - snap = _snapshot() + saved = ( + cells._cell_len, + cells.get_character_cell_size, + cells.split_graphemes, + segment.get_character_cell_size, + ptu._CHAR_SIZES_CACHE, + ew._INSTALLED, + ew._CLUSTERS, + ) + ew._INSTALLED = False + ew.install() try: - ew._CALIBRATED = False - ew.calibrate(force_width=1) - assert cells.cell_len(_WAVE) == 1 - assert cells.cell_len(_DISH) == 2 # unlisted icon stays wide - assert cells.cell_len(_CA) == 2 # flag handled by category, no allowlist entry - assert cells.cell_len(_PLANE) == 2 # wide-VS16 exception carved back out of the narrowing - assert cells.cell_len(_ROAD) == 1 # bare text-default emoji: left exactly as measured - assert cells.cell_len(_WEB) == 1 - assert get_cwidth(_WAVE) == 1 # pt now places the border a cell earlier - assert get_cwidth(_DISH) == 2 - assert get_cwidth(_CA) == 2 # was 4 unpatched - assert get_cwidth(_PLANE) == 2 # was 1 unpatched: the airplane smeared a cell short - assert get_cwidth(_ROAD) == 1 and get_cwidth(_WEB) == 1 # untouched, in both authorities - assert cells.cell_len(_SHRUG) == 2 and get_cwidth(_SHRUG) == 2 # pt counted three - assert cells.cell_len(_FAMILY) == 2 and get_cwidth(_FAMILY) == 2 # pt counted six + yield finally: - _restore(snap) + ( + cells._cell_len, + cells.get_character_cell_size, + cells.split_graphemes, + segment.get_character_cell_size, + ptu._CHAR_SIZES_CACHE, + ew._INSTALLED, + ew._CLUSTERS, + ) = saved + cells.cached_cell_len.cache_clear() + _ANSI_CACHE.clear() + colsnap._CACHE.clear() -def test_calibrate_width2_narrows_nothing_but_still_joins_clusters(monkeypatch) -> None: - """A width-2 terminal narrows nothing, but still gets the cluster join. +def test_whatever_may_be_drawn_as_an_emoji_is_reserved_two_cells(installed) -> None: # noqa: ANN001 + """Every shape of emoji measures two in both authorities, whatever the stock tables said. - Narrowing would itself break the border there — but a joined sequence is - over-measured at *every* width, so that one correction still lands. - - prompt_toolkit sums a ZWJ sequence's codepoints wherever it runs, which no terminal draws: - the family is one two-cell glyph on the widest terminal as surely as on the narrowest. So - the cluster rule is installed on its own here, and nothing else is — the curated sets and - the flag category stay unconfirmed on this renderer, and Rich is not touched at all. + The stock answers were all over the place — prompt_toolkit gave a flag four, a sun asking + for emoji presentation one, a toned thumb four and a family six — and no answer below two + is safe for a glyph a font may draw in two: the pin after it would overwrite its right half. """ - monkeypatch.delenv("MESHTERM_NARROW_EMOJI", raising=False) - from prompt_toolkit.utils import get_cwidth - - snap = _snapshot() - try: - ew._CALIBRATED = False - ew.calibrate(force_width=2) - assert cells.cell_len(_WAVE) == 2 - assert get_cwidth(_WAVE) == 2 - # Rich untouched: it still promotes a VS16 sequence to the two cells this terminal draws. - assert cells.cell_len(_PLANE) == 2 and cells.cell_len("☀️") == 2 - # Neither curated set nor the flag category applies here — pt keeps its stock answers. - assert get_cwidth("☀️") == 1 - assert get_cwidth(_CA) == 4 - # The one correction that holds at either width. - assert get_cwidth(_SHRUG) == 2 # was 3 - assert get_cwidth(_FAMILY) == 2 # was 6 - assert get_cwidth(_TECHIE) == 2 # was 6: a skin tone is no cell, at either width - finally: - _restore(snap) + for glyph in ( + _WAVE, + _DISH, + _SUN, + f"{_SUN}{_VS16}", + _PLANE, + f"{_PLANE}{_VS16}", + _HEART, + _CA, + _CA[0], + _SHRUG, + _FAMILY, + _THUMB, + _TECHIE, + _KEYCAP, + f"{_WARN}{_VS16}", # a mark asking for emoji presentation is an emoji like any other + ): + assert cells.cell_len(glyph) == 2, f"Rich: {glyph!r}" + assert get_cwidth(glyph) == 2, f"prompt_toolkit: {glyph!r}" -def test_a_skin_tone_is_part_of_its_glyph_in_both_authorities() -> None: - """A skin-tone modifier recolours the emoji before it and takes no cell of its own. +def test_text_the_app_draws_keeps_its_stock_width(installed) -> None: # noqa: ANN001 + """The rule reserves nothing it has no reason to: text, chrome and the app's own marks. - wcwidth counts each modifier as two, so prompt_toolkit reserved four cells for ``👍🏽`` and - for the joined ``👨🏻‍💻`` — a two-cell notch in every row naming such a node, and every lane - after the name drawn two columns off. Rich's table already says zero; the cache now agrees, - on both calibration paths. + The marks are text-default emoji too, but they are the app's own vocabulary, drawn in one + cell on the terminals it is used on, so they keep the one cell every screen was laid out + with. Rich's single-cell ranges are measured by a fast path no patch reaches, so everything + has to agree with it there. """ - for narrow, flags in ((frozenset(_WAVE), True), (frozenset(), False)): - cache = ew._make_pt_cache(narrow, frozenset(), flags=flags) - assert cache[_THUMB] == 2 - assert cache[_TECHIE] == 2 - assert cache[f"Tech {_TECHIE} x"] == len("Tech ") + 2 + len(" x") - assert ew._make_cell_len(frozenset(_WAVE), frozenset())(_TECHIE) == 2 + for text, width in ( + ("a", 1), + ("#", 1), + ("1", 1), + ("é", 1), + ("漢", 2), + ("─", 1), + ("⠿", 1), + ("©", 1), + ("▶", 1), + (_WARN, 1), + (_ROAD, 1), + (_WEB, 1), + (_BIN, 1), + ("↕", 1), + ("↔", 1), + ("↩", 1), + ("⌨", 1), + ("⚙", 1), + ("a\U0001f3fd", 1), # a stray skin tone on a letter leaves it a letter + ): + assert cells.cell_len(text) == width, f"Rich: {text!r}" + assert get_cwidth(text) == width, f"prompt_toolkit: {text!r}" -def test_a_stranded_joiner_leaves_the_next_character_its_cell(monkeypatch) -> None: - """A joiner with no pictograph after it joins nothing, in both authorities and both paths. +def test_every_authority_measures_every_string_the_same(installed) -> None: # noqa: ANN001 + """Rich's whole-string width, its grapheme spans and prompt_toolkit's cache all agree. - A MeshCore name is cut at a byte limit, so ``That's So Fetch 🏴‍☠️`` arrives as the flag and - a bare joiner. Both authorities folded whatever came next into the flag — the padding after - the name — so the name lane measured a cell short and every lane after it began a column - before the terminal drew it. + And every lone character measures the same in all of them. These are the four places a width + is read, and a row is only laid out once if they give it one answer. + """ + for text in _SAMPLES: + spans, total = cells.split_graphemes(text) + assert cells.cell_len(text) == total == get_cwidth(text), text + assert total == sum(width for _start, _end, width in spans), text + assert spans[0][0] == 0 and spans[-1][1] == len(text), f"spans cover {text!r}" + for char in text: + size = cells.get_character_cell_size(char) + assert size == segment.get_character_cell_size(char) == get_cwidth(char), repr(char) + + +def test_no_prefix_outgrows_the_character_that_ends_it(installed) -> None: # noqa: ANN001 + """The invariant Rich's segment splitter terminates on, then the splitter itself. + + ``Segment.split_cells`` steps a string a codepoint at a time until a whole-string width meets + the cut, stepping over a two-cell character only where that character measures two on its + own. A prefix that grew by two at a character measuring one would be a cut the walk steps past + in both directions forever — which is why the character width is patched alongside the string + width, and why this checks the invariant before trusting the splitter with it. + """ + for text in _SAMPLES: + for index, char in enumerate(text): + grew = cells.cell_len(text[: index + 1]) - cells.cell_len(text[:index]) + assert 0 <= grew <= 2, f"{text!r} at {index}" + if grew == 2: + assert cells.get_character_cell_size(char) == 2, f"{text!r} at {index}" + + def split_everywhere() -> None: + for text in _SAMPLES: + for cut in range(cells.cell_len(text) + 1): + segment.Segment(text).split_cells(cut) + + worker = threading.Thread(target=split_everywhere, daemon=True) + worker.start() + worker.join(timeout=20) + assert not worker.is_alive(), "a segment split never met its cut" + + +def test_a_stranded_joiner_leaves_the_next_character_its_cell(installed) -> None: # noqa: ANN001 + """A joiner with no pictograph after it joins nothing, in both authorities. + + A MeshCore name is cut at a byte limit, so a name ending on a pirate flag arrives as the + black flag and a bare joiner, followed by the lane's padding. Rich's stock loop folded the + padding into the flag, so the name lane measured a cell short of what the terminal drew. """ - monkeypatch.delenv("MESHTERM_NARROW_EMOJI", raising=False) padded = f"{_STRANDED} x" # the flag's two cells, two of padding, then the next lane - cell_len = ew._make_cell_len(frozenset(_WAVE), frozenset()) - assert cell_len(padded) == 5 - assert ew._make_pt_cache(frozenset(_WAVE), frozenset())[padded] == 5 + assert cells.cell_len(padded) == 5 and get_cwidth(padded) == 5 # A real sequence still joins: the shrug's male sign is a pictograph, not a padding space. - assert cell_len(f"{_SHRUG} x") == 4 + assert cells.cell_len(f"{_SHRUG} x") == 4 and get_cwidth(f"{_SHRUG} x") == 4 - snap = _snapshot() - try: - ew._CALIBRATED = False - ew.calibrate(force_width=2) - # Rich's own loop, trusted on this path, skipped the space and counted four. - assert cells.cell_len(padded) == 5 - assert cells.cell_len(_SHRUG) == 2 and cells.cell_len(_FAMILY) == 2 - finally: - _restore(snap) + +def test_install_happens_once(installed) -> None: # noqa: ANN001 + """A second call is a no-op: the tables patched by the first stay exactly as they are.""" + patched = (cells._cell_len, ptu._CHAR_SIZES_CACHE) + ew.install() + assert (cells._cell_len, ptu._CHAR_SIZES_CACHE) == patched + + +def test_the_app_draws_no_text_default_emoji_its_vocabulary_does_not_name() -> None: + """Every text-default emoji the package draws is one of its own marks, and every mark is used. + + This is what keeps :data:`~meshterm.ui.tui.emoji_width._APP_TEXT_MARKS` a closed vocabulary + rather than a list kept by care: a new one-cell mark added to a screen without being named + there fails here, instead of quietly gaining a blank cell beside it. Prose is skipped + (docstrings and bare strings are read, not drawn), and so is the module that defines the list. + """ + from rich._unicode_data import load + + text_default = { + char + for char in load("auto").narrow_to_wide + if not char.isascii() and char not in cells._SINGLE_CELLS + } + # Inputs to the PicoCalc's substitution table, which maps them to glyphs its font has. They + # are never drawn on a terminal that shows emoji, so they are not marks. + fold_inputs = {"↪", "✔", "✖"} + + package = pathlib.Path(ew.__file__).parents[2] + drawn: set[str] = set() + for path in package.rglob("*.py"): + if path.name == "emoji_width.py": + continue + tree = ast.parse(path.read_text(encoding="utf-8")) + prose = { + id(node.value) + for node in ast.walk(tree) + if isinstance(node, ast.Expr) and isinstance(node.value, ast.Constant) + } + for node in ast.walk(tree): + if isinstance(node, ast.Constant) and isinstance(node.value, str): + if id(node) not in prose: + drawn |= set(node.value) & text_default + + assert drawn - fold_inputs <= ew._APP_TEXT_MARKS, ( + f"name these in _APP_TEXT_MARKS: {sorted(drawn - fold_inputs - ew._APP_TEXT_MARKS)}" + ) + assert ew._APP_TEXT_MARKS <= drawn, ( + f"no longer drawn anywhere: {sorted(ew._APP_TEXT_MARKS - drawn)}" + ) def test_clusters_split_between_glyphs_and_never_inside_one() -> None: - """The unit a lane may cut on: one entry per glyph the terminal actually draws. + """The unit a lane may cut on and a width is measured by: one entry per glyph drawn. - A joined run, a flag's indicator *pair*, and a toned base each come back whole, because - each is one glyph — while a joiner that joins nothing stands on its own, as it does - everywhere else in this module (see :func:`~meshterm.ui.tui.emoji_width._joins`). + A joined run, a flag's indicator *pair*, a toned base and a keycap each come back whole, + because each is one glyph — while a joiner that joins nothing stands on its own (see + :func:`~meshterm.ui.tui.emoji_width._joins`), and so do letters around a joiner. """ - assert list(ew.clusters(f"a{_FAMILY}{_CA}{_THUMB}b")) == ["a", _FAMILY, _CA, _THUMB, "b"] + text = f"a{_FAMILY}{_CA}{_THUMB}{_KEYCAP}b" + assert list(ew.clusters(text)) == ["a", _FAMILY, _CA, _THUMB, _KEYCAP, "b"] # Two flags in a row pair up one at a time rather than running together into four. assert list(ew.clusters(_CA + _CN)) == [_CA, _CN] # A name cut at its byte limit just past a joiner: the joiner is not part of the flag. assert list(ew.clusters(_STRANDED)) == [_FLAG, _ZWJ] + # A joiner between letters joins nothing a font draws as one emoji. + assert list(ew.clusters(f"a{_ZWJ}b")) == ["a", _ZWJ, "b"] def test_cut_cells_keeps_every_glyph_whole_and_strands_no_joiner() -> None: - """A lane's truncation lands between glyphs, so nothing measures what is not drawn. - - The failure this exists for: cutting a codepoint at a time leaves a trailing joiner, - which folds whatever the caller appends — the lane's ellipsis — into the glyph before - it. The ellipsis then measures nothing while the terminal still draws it, and every - column right of the name starts a cell late. - """ + """A lane's truncation lands between glyphs, so nothing is measured that is not drawn.""" assert ew.cut_cells(f"Bob {_FAMILY}", 6) == f"Bob {_FAMILY}" # it fits, so it is kept assert ew.cut_cells(f"Bob {_FAMILY}", 5) == "Bob " # it doesn't: dropped whole assert ew.cut_cells(_CA, 1) == "" # half a flag is a letter, not half a glyph - assert ew.cut_cells(_STRANDED, 4) == _FLAG # the joiner cannot swallow what follows + assert ew.cut_cells(_STRANDED, 4) == _FLAG # the joiner does not trail the cut assert ew.cut_cells("plain name", 5) == "plain" def test_drawable_folds_only_what_no_terminal_can_draw() -> None: """A newline, an escape or a bidi override in an advert name is folded to a space. - None of them is a width bug an allowlist could fix — they are text that must not reach - the terminal at all. The joiner is format-class too and is the one kept, because it is - what holds an emoji sequence together. + None of them is a width question — they are text that must not reach the terminal at all. + The joiner is format-class too and is the one kept, because it holds an emoji together. """ assert ew.drawable("line\nbreak") == "line break" assert ew.drawable("esc\x1b[31mape") == "esc [31mape" - assert ew.drawable("flip‮me") == "flip me" + assert ew.drawable("flip\u202eme") == "flip me" assert ew.drawable(f"Bob {_FAMILY}{_THUMB}") == f"Bob {_FAMILY}{_THUMB}" -def test_join_clusters_merges_only_joined_runs() -> None: - """The fragment merge gathers a whole sequence and leaves everything else alone. +def test_join_clusters_merges_every_multi_codepoint_glyph() -> None: + """The fragment merge gathers each glyph built from several codepoints into one fragment. - prompt_toolkit's ANSI text arrives one codepoint per fragment, so a sequence is a run: - a base, an optional selector, then *joiner + component + optional selector* groups. + prompt_toolkit's ANSI text arrives one codepoint per fragment, so a glyph is a run to be + gathered — and every such run is merged now, not only the joined ones, because a mark + measured one cell and followed by a selector must not be folded into a one-cell slot. """ def line(text: str) -> list: @@ -333,52 +325,41 @@ def test_join_clusters_merges_only_joined_runs() -> None: def texts(fragments: list) -> list: return [text for _style, text in fragments] - # A line with no joiner is handed straight back — the same object, not a copy. - plain = line("hi 📡") + # A line with nothing to merge is handed straight back — the same object, not a copy. + plain = line(f"hi {_DISH} {_WARN}") assert ew._join_clusters(plain) is plain - # The shrug's four fragments become one; the bars around it are untouched. assert texts(ew._join_clusters(line(f"|{_SHRUG}|"))) == ["|", _SHRUG, "|"] - # Three joined people, five fragments, still one glyph. assert texts(ew._join_clusters(line(_FAMILY))) == [_FAMILY] - # A selector on the *base*, before the joiner, belongs to the run: ❤️‍🔥. - burning = "❤️‍\U0001f525" + burning = f"❤{_VS16}{_ZWJ}\U0001f525" # ❤\ufe0f\u200d🔥 a selector on the base, then a join assert texts(ew._join_clusters(line(burning))) == [burning] - # A bare VS16 pair is not a sequence — prompt_toolkit already folds it into the cell - # before it, so those fragments are left exactly as they came. - assert texts(ew._join_clusters(line(f"☀️{_SHRUG}"))) == ["☀", "️", _SHRUG] - # A trailing joiner with nothing to join is not a sequence either. - assert texts(ew._join_clusters(line(f"a{_ZWJ}"))) == ["a", _ZWJ] - # A skin tone on the base belongs to the run the way its selector does. - assert texts(ew._join_clusters(line(f"|{_TECHIE}|"))) == ["|", _TECHIE, "|"] - # A stranded joiner, followed by the name lane's padding, is not a sequence: the space - # after it keeps a fragment — and so a cell — of its own. - assert texts(ew._join_clusters(line(f"{_STRANDED} x"))) == [_FLAG, _ZWJ, " ", "x"] - # A flag's two indicators are one glyph too, written as one: a cursor pin between the - # halves would leave a terminal drawing two letters instead of a flag. + # A selector pair, a toned emoji and a keycap are one glyph each, merged like the rest. + assert texts(ew._join_clusters(line(f"{_SUN}{_VS16}{_SHRUG}"))) == [f"{_SUN}{_VS16}", _SHRUG] + assert texts(ew._join_clusters(line(f"{_WARN}{_VS16}|"))) == [f"{_WARN}{_VS16}", "|"] + assert texts(ew._join_clusters(line(f"{_THUMB}{_KEYCAP}"))) == [_THUMB, _KEYCAP] + # A flag's two indicators are one glyph, written as one: a cursor pin between the halves + # would leave a terminal drawing two letters instead of a flag. assert texts(ew._join_clusters(line(f"|{_CA}|"))) == ["|", _CA, "|"] assert texts(ew._join_clusters(line(f"{_CA}{_CN}"))) == [_CA, _CN] - # A lone indicator is no pair, and stays a fragment of its own. + # A lone indicator is no pair, and a trailing or stranded joiner joins nothing. assert texts(ew._join_clusters(line(f"{_CA[0]} x"))) == [_CA[0], " ", "x"] + assert texts(ew._join_clusters(line(f"a{_ZWJ}"))) == ["a", _ZWJ] + assert texts(ew._join_clusters(line(f"{_STRANDED} x"))) == [_FLAG, _ZWJ, " ", "x"] - # A merged fragment iterates as the whole sequence, which is what makes prompt_toolkit - # build one Char of it instead of one per codepoint. + # A merged fragment iterates as the whole glyph, which is what makes prompt_toolkit build + # one Char of it instead of one per codepoint. (merged,) = texts(ew._join_clusters(line(_SHRUG))) assert list(merged) == [_SHRUG] and merged == _SHRUG -def test_cluster_control_lays_a_sequence_into_a_single_screen_cell(monkeypatch) -> None: - """The delivery half: one ``Char`` per glyph, with every codepoint still written out. +def test_cluster_control_lays_each_glyph_into_a_single_screen_cell(installed) -> None: # noqa: ANN001 + """The delivery half: one two-cell ``Char`` per glyph, every codepoint still written out. - Measuring the sequence right is not enough on its own — prompt_toolkit lays out one - codepoint at a time, so an uncorrected screen puts the male sign in a cell of its own and - everything after it a column late. The control merges the run, so the window makes a - single two-cell ``Char`` of it and the text after lands where the terminal draws it. The - codepoints themselves are untouched: all four still reach the screen, in order, so the - terminal composes the same glyph. + Measuring a glyph right is not enough on its own — prompt_toolkit lays out one codepoint at a + time, so an unmerged shrug puts its male sign in a cell of its own, and a warning mark with a + selector becomes a two-cell glyph in a one-cell slot. Merged, each is one ``Char`` two cells + wide, and the text after lands where the pins will put it. """ - monkeypatch.delenv("MESHTERM_NARROW_EMOJI", raising=False) - monkeypatch.delenv("MESHTERM_WIDE_EMOJI", raising=False) from prompt_toolkit.application import Application from prompt_toolkit.application.current import set_app from prompt_toolkit.formatted_text import ANSI @@ -388,30 +369,18 @@ def test_cluster_control_lays_a_sequence_into_a_single_screen_cell(monkeypatch) from prompt_toolkit.layout.screen import Screen, WritePosition from prompt_toolkit.output import DummyOutput - text = f"|{_SHRUG}|{_FAMILY}|end" + warn = f"{_WARN}{_VS16}" + text = f"|{_SHRUG}|{_FAMILY}|{warn}|end" + window = Window(ew.ClusterTextControl(lambda: ANSI(text)), always_hide_cursor=True) + app = Application(layout=Layout(window), input=DummyInput(), output=DummyOutput()) + with set_app(app): + screen = Screen(default_char=None, initial_width=40, initial_height=1) + window.write_to_screen(screen, MouseHandlers(), WritePosition(0, 0, 40, 1), "", False, None) + row = screen.data_buffer[0] - def paint() -> list: - window = Window(ew.ClusterTextControl(lambda: ANSI(text)), always_hide_cursor=True) - app = Application(layout=Layout(window), input=DummyInput(), output=DummyOutput()) - with set_app(app): - screen = Screen(default_char=None, initial_width=40, initial_height=1) - window.write_to_screen( - screen, MouseHandlers(), WritePosition(0, 0, 40, 1), "", False, None - ) - return screen.data_buffer[0] - - snap = _snapshot() - try: - ew._CALIBRATED = False - ew.calibrate(force_width=1) - row = paint() - # Each sequence is one cell-occupying Char, two cells wide, exactly as drawn. - assert (row[1].char, row[1].width) == (_SHRUG, 2) - assert (row[4].char, row[4].width) == (_FAMILY, 2) - # So the text after them starts at column 6 rather than 13 — seven cells of border - # drift removed (one for the shrug, four for the family, two from the second bar). - assert "".join(row[x].char for x in range(6, 10)) == "|end" - # Nothing was dropped on the way: the row still spells the source exactly. - assert "".join(row[x].char for x in range(40)).rstrip() == text - finally: - _restore(snap) + assert (row[1].char, row[1].width) == (_SHRUG, 2) + assert (row[4].char, row[4].width) == (_FAMILY, 2) + assert (row[7].char, row[7].width) == (warn, 2) + assert "".join(row[x].char for x in range(9, 13)) == "|end" + # Nothing was dropped on the way: the row still spells the source exactly. + assert "".join(row[x].char for x in range(40)).rstrip() == text diff --git a/tests/test_win32dll.py b/tests/test_win32dll.py index c00e4f2..04fd26b 100644 --- a/tests/test_win32dll.py +++ b/tests/test_win32dll.py @@ -66,17 +66,19 @@ def test_declaring_a_signature_does_not_reach_the_shared_one() -> None: assert private.GetConsoleScreenBufferInfo is not shared -def test_the_probe_leaves_the_shared_signature_alone() -> None: - """End to end: running the emoji probe must not disturb the process-wide function. +def test_the_console_probe_leaves_the_shared_signature_alone() -> None: + """End to end: a real call site declaring the function must not disturb the shared one. - The narrow assertion above can pass while the real call site still reaches for - ``ctypes.windll``, so this exercises the code that actually crashed. + The narrow assertion above can pass while a call site still reaches for + ``ctypes.windll``, so this exercises one. The emoji-width probe that crashed is gone, but + the classic-console check declares the very same ``GetConsoleScreenBufferInfo`` with a + struct of its own, which is the same crash waiting in a different module. """ - from meshterm.ui.tui import emoji_width + from meshterm.ui import termfont shared = ctypes.windll.kernel32.GetConsoleScreenBufferInfo before = shared.argtypes - # No console under pytest, so this returns None rather than measuring. That is fine: + # No console under pytest, so this returns None rather than answering. That is fine: # the declarations happen before the first call either way, which is what matters. - emoji_width._win_probe_width("x") # noqa: SLF001 - the point is this private path + termfont._is_classic_console() # noqa: SLF001 - the point is this private path assert ctypes.windll.kernel32.GetConsoleScreenBufferInfo.argtypes == before