diff --git a/CLAUDE.md b/CLAUDE.md index 8b74a74..180fc76 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -456,6 +456,11 @@ as a grouped list does. Filling a page in is editing its `.md`; no Python follow long for its hanging lane steps out to the gutter on a line of its own (`render_hanging(whole=…, gutter=…)`), since a terminal opens a link, and a reader copies one, only while it sits on one line. Only a URL wider than the screen folds. + **A link is what `core/links.py` finds**, and nothing guesses beside it: a scheme + (`scheme://…`), or a host written without one where nothing else could be meant — + starting `www.`, or ending in a TLD IANA has delegated (`assets/tlds.txt`, the registry + verbatim; refreshing it is downloading it again). So `meshterm.net` is a link and + `file.txt` is not. A scheme-less link opens, and encodes, as `https://…`. - Dialogs anchor slightly above true centre, sized for their populated state. - Radio traffic: single transmissions or a user-chosen sample count with cooldown pacing — never bursts. diff --git a/meshterm/assets/tlds.txt b/meshterm/assets/tlds.txt new file mode 100644 index 0000000..f6957b5 --- /dev/null +++ b/meshterm/assets/tlds.txt @@ -0,0 +1,1438 @@ +# Version 2026093003, Last Updated Thu Oct 1 07:07:01 2026 UTC +AAA +AARP +ABB +ABBOTT +ABBVIE +ABC +ABLE +ABOGADO +ABUDHABI +AC +ACADEMY +ACCENTURE +ACCOUNTANT +ACCOUNTANTS +ACO +ACTOR +AD +ADS +ADULT +AE +AEG +AERO +AETNA +AF +AFL +AFRICA +AG +AGAKHAN +AGENCY +AI +AIG +AIRBUS +AIRFORCE +AIRTEL +AKDN +AL +ALIBABA +ALIPAY +ALLFINANZ +ALLSTATE +ALLY +ALSACE +ALSTOM +AM +AMAZON +AMERICANEXPRESS +AMERICANFAMILY +AMEX +AMFAM +AMICA +AMSTERDAM +ANALYTICS +ANDROID +ANQUAN +ANZ +AO +AOL +APARTMENTS +APP +APPLE +AQ +AQUARELLE +AR +ARAB +ARAMCO +ARCHI +ARMY +ARPA +ART +ARTE +AS +ASDA +ASIA +ASSOCIATES +AT +ATHLETA +ATTORNEY +AU +AUCTION +AUDI +AUDIBLE +AUDIO +AUSPOST +AUTHOR +AUTO +AUTOS +AW +AWS +AX +AXA +AZ +AZURE +BA +BABY +BAIDU +BANAMEX +BAND +BANK +BAR +BARCELONA +BARCLAYCARD +BARCLAYS +BAREFOOT +BARGAINS +BASEBALL +BASKETBALL +BAUHAUS +BAYERN +BB +BBC +BBT +BBVA +BCG +BCN +BD +BE +BEATS +BEAUTY +BEER +BERLIN +BEST +BESTBUY +BET +BF +BG +BH +BHARTI +BI +BIBLE +BID +BIKE +BING +BINGO +BIO +BIZ +BJ +BLACK +BLACKFRIDAY +BLOCKBUSTER +BLOG +BLOOMBERG +BLUE +BM +BMS +BMW +BN +BNPPARIBAS +BO +BOATS +BOEHRINGER +BOFA +BOM +BOND +BOO +BOOK +BOOKING +BOSCH +BOSTIK +BOSTON +BOT +BOUTIQUE +BOX +BR +BRADESCO +BRIDGESTONE +BROADWAY +BROKER +BROTHER +BRUSSELS +BS +BT +BUILD +BUILDERS +BUSINESS +BUY +BUZZ +BV +BW +BY +BZ +BZH +CA +CAB +CAFE +CAL +CALL +CALVINKLEIN +CAM +CAMERA +CAMP +CANON +CAPETOWN +CAPITAL +CAPITALONE +CAR +CARAVAN +CARDS +CARE +CAREER +CAREERS +CARS +CASA +CASE +CASH +CASINO +CAT +CATERING +CATHOLIC +CBA +CBN +CBRE +CC +CD +CENTER +CEO +CERN +CF +CFA +CFD +CG +CH +CHANEL +CHANNEL +CHARITY +CHASE +CHAT +CHEAP +CHINTAI +CHRISTMAS +CHROME +CHURCH +CI +CIPRIANI +CIRCLE +CISCO +CITADEL +CITI +CITIC +CITY +CK +CL +CLAIMS +CLEANING +CLICK +CLINIC +CLINIQUE +CLOTHING +CLOUD +CLUB +CLUBMED +CM +CN +CO +COACH +CODES +COFFEE +COLLEGE +COLOGNE +COM +COMMBANK +COMMUNITY +COMPANY +COMPARE +COMPUTER +COMSEC +CONDOS +CONSTRUCTION +CONSULTING +CONTACT +CONTRACTORS +COOKING +COOL +COOP +CORSICA +COUNTRY +COUPON +COUPONS +COURSES +CPA +CR +CREDIT +CREDITCARD +CREDITUNION +CRICKET +CROWN +CRS +CRUISE +CRUISES +CU +CUISINELLA +CV +CW +CX +CY +CYMRU +CYOU +CZ +DAD +DANCE +DATA +DATE +DATING +DATSUN +DAY +DCLK +DDS +DE +DEAL +DEALER +DEALS +DEGREE +DELIVERY +DELL +DELOITTE +DELTA +DEMOCRAT +DENTAL +DENTIST +DESI +DESIGN +DEV +DHL +DIAMONDS +DIET +DIGITAL +DIRECT +DIRECTORY +DISCOUNT +DISCOVER +DISH +DIY +DJ +DK +DM +DNP +DO +DOCS +DOCTOR +DOG +DOMAINS +DOT +DOWNLOAD +DRIVE +DTV +DUBAI +DUPONT +DURBAN +DVAG +DVR +DZ +EARTH +EAT +EC +ECO +EDEKA +EDU +EDUCATION +EE +EG +EMAIL +EMERCK +ENERGY +ENGINEER +ENGINEERING +ENTERPRISES +EPSON +EQUIPMENT +ER +ERICSSON +ERNI +ES +ESQ +ESTATE +ET +EU +EUROVISION +EUS +EVENTS +EXCHANGE +EXPERT +EXPOSED +EXPRESS +EXTRASPACE +FAGE +FAIL +FAIRWINDS +FAITH +FAMILY +FAN +FANS +FARM +FARMERS +FASHION +FAST +FEDEX +FEEDBACK +FERRARI +FERRERO +FI +FIDELITY +FIDO +FILM +FINAL +FINANCE +FINANCIAL +FIRE +FIRESTONE +FIRMDALE +FISH +FISHING +FIT +FITNESS +FJ +FK +FLICKR +FLIGHTS +FLIR +FLORIST +FLOWERS +FLY +FM +FO +FOO +FOOD +FOOTBALL +FORD +FOREX +FORSALE +FORUM +FOUNDATION +FOX +FR +FREE +FRESENIUS +FRL +FROGANS +FRONTIER +FTR +FUJITSU +FUN +FUND +FURNITURE +FUTBOL +FYI +GA +GAL +GALLERY +GALLO +GALLUP +GAME +GAMES +GAP +GARDEN +GAY +GB +GBIZ +GD +GDN +GE +GEA +GENT +GENTING +GEORGE +GF +GG +GGEE +GH +GI +GIFT +GIFTS +GIVES +GIVING +GL +GLASS +GLE +GLOBAL +GLOBO +GM +GMAIL +GMBH +GMO +GMX +GN +GODADDY +GOLD +GOLDPOINT +GOLF +GOODYEAR +GOOG +GOOGLE +GOP +GOT +GOV +GP +GQ +GR +GRAINGER +GRAPHICS +GRATIS +GREEN +GRIPE +GROCERY +GROUP +GS +GT +GU +GUCCI +GUGE +GUIDE +GUITARS +GURU +GW +GY +HAIR +HAMBURG +HANGOUT +HAUS +HBO +HDFC +HDFCBANK +HEALTH +HEALTHCARE +HELP +HELSINKI +HERE +HERMES +HIPHOP +HISAMITSU +HITACHI +HIV +HK +HKT +HM +HN +HOCKEY +HOLDINGS +HOLIDAY +HOMEDEPOT +HOMEGOODS +HOMES +HOMESENSE +HONDA +HORSE +HOSPITAL +HOST +HOSTING +HOT +HOTELS +HOTMAIL +HOUSE +HOW +HR +HSBC +HT +HU +HUGHES +HYATT +HYUNDAI +IBM +ICBC +ICE +ICU +ID +IE +IEEE +IFM +IKANO +IL +IM +IMAMAT +IMDB +IMMO +IMMOBILIEN +IN +INC +INDUSTRIES +INFINITI +INFO +ING +INK +INSTITUTE +INSURANCE +INSURE +INT +INTERNATIONAL +INTUIT +INVESTMENTS +IO +IPIRANGA +IQ +IR +IRISH +IS +ISMAILI +IST +ISTANBUL +IT +ITAU +ITV +JAGUAR +JAVA +JCB +JE +JEEP +JETZT +JEWELRY +JIO +JLL +JM +JMP +JNJ +JO +JOBS +JOBURG +JOT +JOY +JP +JPMORGAN +JPRS +JUEGOS +KAUFEN +KDDI +KE +KERRYHOTELS +KERRYPROPERTIES +KFH +KG +KH +KI +KIA +KIDS +KIM +KINDLE +KITCHEN +KIWI +KM +KN +KOELN +KOMATSU +KOSHER +KP +KPMG +KPN +KR +KRD +KRED +KUOKGROUP +KW +KY +KYOTO +KZ +LA +LACAIXA +LAMBORGHINI +LAMER +LAND +LANDROVER +LANXESS +LASALLE +LAT +LATINO +LATROBE +LAW +LAWYER +LB +LC +LDS +LEASE +LECLERC +LEFRAK +LEGAL +LEGO +LEXUS +LGBT +LI +LIDL +LIFE +LIFEINSURANCE +LIFESTYLE +LIGHTING +LIKE +LILLY +LIMITED +LIMO +LINCOLN +LINK +LIVE +LIVING +LK +LLC +LLP +LOAN +LOANS +LOCKER +LOCUS +LOL +LONDON +LOTTE +LOTTO +LOVE +LPL +LPLFINANCIAL +LR +LS +LT +LTD +LTDA +LU +LUNDBECK +LUXE +LUXURY +LV +LY +MA +MADRID +MAIF +MAISON +MAKEUP +MAN +MANAGEMENT +MANGO +MAP +MARKET +MARKETING +MARKETS +MARRIOTT +MARSHALLS +MATTEL +MBA +MC +MCKINSEY +MD +ME +MED +MEDIA +MEET +MELBOURNE +MEME +MEMORIAL +MEN +MENU +MERCK +MERCKMSD +MG +MH +MIAMI +MICROSOFT +MIL +MINI +MINT +MIT +MITSUBISHI +MK +ML +MLB +MLS +MM +MMA +MN +MO +MOBI +MOBILE +MODA +MOE +MOI +MOM +MONASH +MONEY +MONSTER +MORMON +MORTGAGE +MOSCOW +MOTO +MOTORCYCLES +MOV +MOVIE +MP +MQ +MR +MS +MSD +MT +MTN +MTR +MU +MUSEUM +MUSIC +MV +MW +MX +MY +MZ +NA +NAB +NAGOYA +NAME +NAVY +NBA +NC +NE +NEC +NET +NETBANK +NETFLIX +NETWORK +NEUSTAR +NEW +NEWS +NEXT +NEXTDIRECT +NEXUS +NF +NFL +NG +NGO +NHK +NI +NICO +NIKE +NIKON +NINJA +NISSAN +NISSAY +NL +NO +NOKIA +NORTON +NOW +NOWRUZ +NOWTV +NP +NR +NRA +NRW +NTT +NU +NYC +NZ +OBI +OBSERVER +OFFICE +OKINAWA +OLAYAN +OLAYANGROUP +OLLO +OM +OMEGA +ONE +ONG +ONL +ONLINE +OOO +OPEN +ORACLE +ORANGE +ORG +ORGANIC +ORIGINS +OSAKA +OTSUKA +OTT +OVH +PA +PAGE +PANASONIC +PARIS +PARS +PARTNERS +PARTS +PARTY +PAY +PCCW +PE +PET +PF +PFIZER +PG +PH +PHARMACY +PHD +PHILIPS +PHONE +PHOTO +PHOTOGRAPHY +PHOTOS +PHYSIO +PICS +PICTET +PICTURES +PID +PIN +PING +PINK +PIONEER +PIZZA +PK +PL +PLACE +PLAY +PLAYSTATION +PLUMBING +PLUS +PM +PN +PNC +POHL +POKER +POLITIE +PORN +POST +PR +PRAXI +PRESS +PRIME +PRO +PROD +PRODUCTIONS +PROF +PROGRESSIVE +PROMO +PROPERTIES +PROPERTY +PROTECTION +PRU +PRUDENTIAL +PS +PT +PUB +PW +PWC +PY +QA +QPON +QUEBEC +QUEST +RACING +RADIO +RE +READ +REALESTATE +REALTOR +REALTY +RECIPES +RED +REDUMBRELLA +REHAB +REISE +REISEN +REIT +RELIANCE +REN +RENT +RENTALS +REPAIR +REPORT +REPUBLICAN +REST +RESTAURANT +REVIEW +REVIEWS +REXROTH +RICH +RICHARDLI +RICOH +RIL +RIO +RIP +RO +ROCKS +RODEO +ROGERS +ROOM +RS +RSVP +RU +RUGBY +RUHR +RUN +RW +RWE +RYUKYU +SA +SAARLAND +SAFE +SAFETY +SAKURA +SALE +SALON +SAMSCLUB +SAMSUNG +SANDVIK +SANDVIKCOROMANT +SANOFI +SAP +SARL +SAS +SAVE +SAXO +SB +SBI +SBS +SC +SCB +SCHAEFFLER +SCHMIDT +SCHOLARSHIPS +SCHOOL +SCHULE +SCHWARZ +SCIENCE +SCOT +SD +SE +SEARCH +SEAT +SECURE +SECURITY +SEEK +SELECT +SENER +SERVICES +SEVEN +SEW +SEX +SEXY +SFR +SG +SH +SHANGRILA +SHARP +SHELL +SHIA +SHIKSHA +SHOES +SHOP +SHOPPING +SHOUJI +SHOW +SI +SILK +SINA +SINGLES +SITE +SJ +SK +SKI +SKIN +SKY +SKYPE +SL +SLING +SM +SMART +SMILE +SN +SNCF +SO +SOCCER +SOCIAL +SOFTBANK +SOFTWARE +SOHU +SOLAR +SOLUTIONS +SONG +SONY +SOY +SPA +SPACE +SPORT +SPOT +SR +SRL +SS +ST +STADA +STAPLES +STAR +STATEBANK +STATEFARM +STC +STCGROUP +STOCKHOLM +STORAGE +STORE +STREAM +STUDIO +STUDY +STYLE +SU +SUCKS +SUPPLIES +SUPPLY +SUPPORT +SURF +SURGERY +SUZUKI +SV +SWATCH +SWISS +SX +SY +SYDNEY +SYSTEMS +SZ +TAB +TAIPEI +TALK +TAOBAO +TARGET +TATAMOTORS +TATAR +TATTOO +TAX +TAXI +TC +TCI +TD +TDK +TEAM +TECH +TECHNOLOGY +TEL +TEMASEK +TENNIS +TEVA +TF +TG +TH +THD +THEATER +THEATRE +TIAA +TICKETS +TIENDA +TIPS +TIRES +TIROL +TJ +TJMAXX +TJX +TK +TKMAXX +TL +TM +TMALL +TN +TO +TODAY +TOKYO +TOOLS +TOP +TORAY +TOSHIBA +TOTAL +TOURS +TOWN +TOYOTA +TOYS +TR +TRADE +TRADING +TRAINING +TRAVEL +TRAVELERS +TRAVELERSINSURANCE +TRUST +TRV +TT +TUBE +TUI +TUNES +TUSHU +TV +TVS +TW +TZ +UA +UBANK +UBS +UG +UK +UNICOM +UNIVERSITY +UNO +UOL +UPS +US +UY +UZ +VA +VACATIONS +VANA +VANGUARD +VC +VE +VEGAS +VENTURES +VERISIGN +VERSICHERUNG +VET +VG +VI +VIAJES +VIDEO +VIG +VIKING +VILLAS +VIN +VIP +VIRGIN +VISA +VISION +VIVA +VIVO +VLAANDEREN +VN +VODKA +VOLVO +VOTE +VOTING +VOTO +VOYAGE +VU +WALES +WALMART +WALTER +WANG +WANGGOU +WATCH +WATCHES +WEATHER +WEATHERCHANNEL +WEB +WEBCAM +WEBER +WEBSITE +WED +WEDDING +WEIBO +WEIR +WF +WHOSWHO +WIEN +WIKI +WILLIAMHILL +WIN +WINDOWS +WINE +WINNERS +WME +WOODSIDE +WORK +WORKS +WORLD +WOW +WS +WTC +WTF +XBOX +XEROX +XIHUAN +XIN +XN--11B4C3D +XN--1CK2E1B +XN--1QQW23A +XN--2SCRJ9C +XN--30RR7Y +XN--3BST00M +XN--3DS443G +XN--3E0B707E +XN--3HCRJ9C +XN--3PXU8K +XN--42C2D9A +XN--45BR5CYL +XN--45BRJ9C +XN--45Q11C +XN--4DBRK0CE +XN--4GBRIM +XN--54B7FTA0CC +XN--55QW42G +XN--55QX5D +XN--5SU34J936BGSG +XN--5TZM5G +XN--6FRZ82G +XN--6QQ986B3XL +XN--80ADXHKS +XN--80AO21A +XN--80AQECDR1A +XN--80ASEHDB +XN--80ASWG +XN--8Y0A063A +XN--90A3AC +XN--90AE +XN--90AIS +XN--9DBQ2A +XN--9ET52U +XN--9KRT00A +XN--B4W605FERD +XN--BCK1B9A5DRE4C +XN--C1AVG +XN--C2BR7G +XN--CCK2B3B +XN--CCKWCXETD +XN--CG4BKI +XN--CLCHC0EA0B2G2A9GCD +XN--CZR694B +XN--CZRS0T +XN--CZRU2D +XN--D1ACJ3B +XN--D1ALF +XN--E1A4C +XN--ECKVDTC9D +XN--EFVY88H +XN--FCT429K +XN--FHBEI +XN--FIQ228C5HS +XN--FIQ64B +XN--FIQS8S +XN--FIQZ9S +XN--FJQ720A +XN--FLW351E +XN--FPCRJ9C3D +XN--FZC2C9E2C +XN--FZYS8D69UVGM +XN--G2XX48C +XN--GCKR3F0F +XN--GECRJ9C +XN--GK3AT1E +XN--H2BREG3EVE +XN--H2BRJ9C +XN--H2BRJ9C8C +XN--HXT814E +XN--I1B6B1A6A2E +XN--IMR513N +XN--IO0A7I +XN--J1AEF +XN--J1AMH +XN--J6W193G +XN--JLQ480N2RG +XN--JVR189M +XN--KCRX77D1X4A +XN--KPRW13D +XN--KPRY57D +XN--KPUT3I +XN--L1ACC +XN--LGBBAT1AD8J +XN--MGB9AWBF +XN--MGBA3A3EJT +XN--MGBA3A4F16A +XN--MGBA7C0BBN0A +XN--MGBAAM7A8H +XN--MGBAB2BD +XN--MGBAH1A3HJKRD +XN--MGBAI9AZGQP6J +XN--MGBAYH7GPA +XN--MGBBH1A +XN--MGBBH1A71E +XN--MGBC0A9AZCG +XN--MGBCA7DZDO +XN--MGBCPQ6GPA1A +XN--MGBERP4A5D4AR +XN--MGBGU82A +XN--MGBI4ECEXP +XN--MGBPL2FH +XN--MGBT3DHD +XN--MGBTX2B +XN--MGBX4CD0AB +XN--MIX891F +XN--MK1BU44C +XN--MXTQ1M +XN--NGBC5AZD +XN--NGBE9E0A +XN--NGBRX +XN--NODE +XN--NQV7F +XN--NQV7FS00EMA +XN--NYQY26A +XN--O3CW4H +XN--OGBPF8FL +XN--OTU796D +XN--P1ACF +XN--P1AI +XN--PGBS0DH +XN--PSSY2U +XN--Q7CE6A +XN--Q9JYB4C +XN--QCKA1PMC +XN--QXA6A +XN--QXAM +XN--RHQV96G +XN--ROVU88B +XN--RVC1E0AM3E +XN--S9BRJ9C +XN--SES554G +XN--T60B56A +XN--TCKWE +XN--TIQ49XQYJ +XN--UNUP4Y +XN--VERMGENSBERATER-CTB +XN--VERMGENSBERATUNG-PWB +XN--VHQUV +XN--VUQ861B +XN--W4R85EL8FHU5DNRA +XN--W4RS40L +XN--WGBH1C +XN--WGBL6A +XN--XHQ521B +XN--XKC2AL3HYE2A +XN--XKC2DL3A5EE0H +XN--Y9A3AQ +XN--YFRO4I67O +XN--YGBI2AMMX +XN--ZFR164B +XXX +XYZ +YACHTS +YAHOO +YAMAXUN +YANDEX +YE +YODOBASHI +YOGA +YOKOHAMA +YOU +YOUTUBE +YT +YUN +ZA +ZAPPOS +ZARA +ZERO +ZIP +ZM +ZONE +ZUERICH +ZW diff --git a/meshterm/core/links.py b/meshterm/core/links.py new file mode 100644 index 0000000..bd27c20 --- /dev/null +++ b/meshterm/core/links.py @@ -0,0 +1,104 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Find the links in a line of text — written with a scheme, or without one where it is plain. + +A link with a scheme is anything ``scheme://…``: a scheme rather than a list of them, so a +``meshcore://`` share pasted into a chat is found as surely as a web link is. One written +without a scheme is accepted only where nothing else could be meant — a host starting +``www.``, or one whose last label is a top-level domain IANA has delegated +(``meshterm.net``, ``github.com/…``). So ``file.txt`` is not a link, ``txt`` being no TLD, +and neither is ``3.14``, ``e.g.`` or an address like ``jp@meshterm.net``: a host only +counts where a word begins, never inside an address, a path or another host. + +The registry is IANA's own list, kept verbatim in ``assets/tlds.txt`` with its version +line on top; refreshing it is downloading the file again +(https://data.iana.org/TLD/tlds-alpha-by-domain.txt). Some delegated TLDs are also file +extensions — ``.py``, ``.md``, ``.zip`` — and a name ending in one reads as a link here. +That is the registry's answer rather than a guess, and the cost of it is a code offered +for a link nobody meant, which is the cheaper way to be wrong. + +Every link is handed back as the URL it opens: one written without a scheme gains +``https://``, because a phone handed a bare host in a QR code reads it as text. +""" + +from __future__ import annotations + +import re +from functools import cache +from pathlib import Path + +#: IANA's list of delegated top-level domains, one per line, upper case, ``#`` comments. +TLDS_FILE = Path(__file__).resolve().parent.parent / "assets" / "tlds.txt" + +#: One host label: letters, digits and inner hyphens, at most 63 long. +_LABEL = r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?" + +#: A link with a scheme: the scheme, ``://``, and everything up to the next space. +_SCHEMED = r"\b[a-z][a-z0-9+.-]*://[^\s<>\"]+" + +#: A host written without a scheme, then an optional port and path. It must start a word — +#: not follow a letter, a digit, ``@`` (an address), ``/`` or ``:`` (a path), or a ``.`` +#: or ``-`` (the middle of a longer name) — and its last label must be all letters, the +#: shape of every TLD, so a version number or a decimal never gets this far. +_BARE = ( + r"(?(?:{_LABEL}\.)+(?P[a-z]{{2,63}}))(?![\w-])" + r"(?::\d{1,5})?" + r"(?:[/?#][^\s<>\"]*)?" +) + +#: Either kind. The scheme comes first, so a host inside a scheme's link is consumed with +#: it and never found a second time on its own. +_LINK = re.compile(rf"(?P{_SCHEMED})|(?P{_BARE})", re.IGNORECASE) + +#: Punctuation that follows a link in a sentence and isn't part of it. +_TRAILING = ".,;:!?'\"" + +#: Each closing bracket and its opener: a closer ends a link only when it has no opener +#: inside it, so ``(see https://example.com/a_(b))`` keeps the ``)`` that belongs to it. +_CLOSERS = {")": "(", "]": "[", "}": "{"} + + +@cache +def tlds() -> frozenset[str]: + """Every delegated top-level domain, lower case — read once, on the first link looked for.""" + lines = TLDS_FILE.read_text(encoding="ascii").splitlines() + return frozenset(line.strip().lower() for line in lines if line.strip()[:1] not in ("", "#")) + + +def _trimmed(link: str) -> str: + """``link`` without the sentence's punctuation after it, or a bracket it never opened.""" + while link: + last = link[-1] + if last in _TRAILING or ( + last in _CLOSERS and link.count(last) > link.count(_CLOSERS[last]) + ): + link = link[:-1] + else: + break + return link + + +def urls(text: str) -> list[str]: + """Every link in ``text`` as the URL it opens, in reading order, each only once. + + Trailing punctuation is the sentence's, not the link's; a link written without a + scheme comes back with ``https://`` in front of it, so ``meshterm.net`` and + ``https://meshterm.net`` in one message are the same link, found once. + """ + found: list[str] = [] + for match in _LINK.finditer(text): + link = _trimmed(match.group()) + if match.group("scheme"): + if link.endswith("://"): + continue # a scheme and nothing after it + else: + host = match.group("host").lower() + if not host.startswith("www.") and match.group("tld").lower() not in tlds(): + continue + link = "https://" + link + if link not in found: + found.append(link) + return found + + +__all__ = ["TLDS_FILE", "tlds", "urls"] diff --git a/meshterm/ui/chat.py b/meshterm/ui/chat.py index 3f45df1..a1df401 100644 --- a/meshterm/ui/chat.py +++ b/meshterm/ui/chat.py @@ -24,6 +24,7 @@ from typing import TYPE_CHECKING, Any from rich.console import Group, RenderableType from rich.text import Text +from ..core import links from ..core.channels import MENTION, split_channel_sender from ..core.connection import ContactNotOnDeviceError from ..core.events import EventKind, MeshEvent @@ -55,18 +56,6 @@ if TYPE_CHECKING: _DELIVERED = ("✓", "ok") _FAILED = ("✗", "err") -#: A URL in a message body: a scheme, ``://``, and everything up to the next space. A -#: scheme rather than a list of them, so a ``meshcore://`` share pasted into a chat gets -#: its code as surely as a web link does. -_URL = re.compile(r"\b[a-z][a-z0-9+.-]*://[^\s<>\"]+", re.IGNORECASE) - -#: Punctuation that follows a URL in a sentence and isn't part of it. -_TRAILING = ".,;:!?'\"" - -#: Each closing bracket and its opener: a closer ends a URL only when it has no opener -#: inside it, so ``(see https://example.com/a_(b))`` keeps the ``)`` that belongs to it. -_CLOSERS = {")": "(", "]": "[", "}": "{"} - #: A whitespace-delimited word — the unit the wrap moves whole, so the run a URL must not be #: cut across is the word it sits in, any bracket or full stop beside it included. _WORD = re.compile(r"\S+") @@ -87,24 +76,6 @@ def _bind(platform: Platform) -> None: _URL_CODES = platform.url_codes -def _urls(body: str) -> list[str]: - """Every URL in ``body`` in reading order, each only once, without trailing punctuation.""" - found: list[str] = [] - for match in _URL.finditer(body): - url = match.group() - while url: - last = url[-1] - if last in _TRAILING or ( - last in _CLOSERS and url.count(last) > url.count(_CLOSERS[last]) - ): - url = url[:-1] - else: - break - if not url.endswith("://") and url not in found: - found.append(url) - return found - - class ChatScreen(Screen): """A live conversation: a scrolling transcript above a pinned input line. @@ -609,11 +580,11 @@ class ChatScreen(Screen): prefix.append("❯ " if selected else " ", style="cursor" if selected else None) prefix.append(f"{stamp:%H:%M} ", style="cursor" if selected else "muted") body_text = self._body_text(body, message, selected=selected) - whole = [m.span() for m in _WORD.finditer(body_text.plain) if _URL.search(m.group())] + whole = [m.span() for m in _WORD.finditer(body_text.plain) if links.urls(m.group())] lines = render_hanging( prefix, body_text, width, indent=prefix.cell_len, whole=whole, gutter=_URL_GUTTER ) - urls = _urls(body) if _URL_CODES else [] + urls = links.urls(body) if _URL_CODES else [] if urls: lines += render_lines(qr_strip(urls, width, indent=prefix.cell_len), width) return lines @@ -836,7 +807,7 @@ class ChatScreen(Screen): if self._selected is None or not self._messages: return [] message = self._messages[max(0, min(self._selected, len(self._messages) - 1))] - return _urls(self._sender_and_body(message)[1]) + return links.urls(self._sender_and_body(message)[1]) def _open_codes(self) -> None: """Show the picked message's links as QR codes on the share screen (^U). diff --git a/tests/test_chat.py b/tests/test_chat.py index bb618d6..8bfc160 100644 --- a/tests/test_chat.py +++ b/tests/test_chat.py @@ -2332,23 +2332,6 @@ def _decode_braille(rows: list[str], left: int, size: int) -> list[list[bool]]: ] -def test_urls_are_found_whole_once_and_without_the_sentence_around_them() -> None: - """A URL ends where its sentence's punctuation begins, and keeps a bracket it opened.""" - from meshterm.ui.chat import _urls - - body = ( - "map at https://meshterm.net. see (https://en.wikipedia.org/wiki/Foo_(bar)), " - "'https://example.org/a?b=1' and meshcore://channel/add?name=Ops&secret=ab — " - "again https://meshterm.net! nothing here: https:// or note:x" - ) - assert _urls(body) == [ - "https://meshterm.net", - "https://en.wikipedia.org/wiki/Foo_(bar)", - "https://example.org/a?b=1", - "meshcore://channel/add?name=Ops&secret=ab", - ] - - def test_on_the_picocalc_each_url_hangs_a_code_under_its_message_side_by_side() -> None: """Two links, two braille codes on the same rows, hung at the body's own indent. @@ -2530,3 +2513,24 @@ def test_ctrl_u_is_named_only_on_a_picked_message_with_a_link() -> None: for screen in (linked, direct, scoped): assert "^U QR" in screen.footer_hint assert len(screen.footer_hint) <= 72, screen.footer_hint + + +def test_a_link_written_without_a_scheme_gets_its_code_too() -> None: + """``meshterm.net/map`` is as much a link as its https form, and opens as that.""" + from meshterm.ui.qr import QrScreen + + opened: list = [] + + class Session(_StubSession): + async def run_screen(self, screen): # noqa: ANN001, ANN202 + opened.append(screen) + + async def scenario() -> None: + screen = _url_chat("map at meshterm.net/map, notes in file.txt", Session()) + screen.handle("up") + assert "^U QR" in screen.footer_hint + screen.handle("url_code") + await asyncio.sleep(0) + assert isinstance(opened[0], QrScreen) and opened[0].urls == ("https://meshterm.net/map",) + + asyncio.run(scenario()) diff --git a/tests/test_links.py b/tests/test_links.py new file mode 100644 index 0000000..199ab3a --- /dev/null +++ b/tests/test_links.py @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Finding links in a line of text — with a scheme, and without one where it is plain.""" + +from __future__ import annotations + +from meshterm.core.links import TLDS_FILE, tlds, urls + + +def test_urls_are_found_whole_once_and_without_the_sentence_around_them() -> None: + """A URL ends where its sentence's punctuation begins, and keeps a bracket it opened.""" + body = ( + "map at https://meshterm.net. see (https://en.wikipedia.org/wiki/Foo_(bar)), " + "'https://example.org/a?b=1' and meshcore://channel/add?name=Ops&secret=ab — " + "again https://meshterm.net! nothing here: https:// or note:x" + ) + assert urls(body) == [ + "https://meshterm.net", + "https://en.wikipedia.org/wiki/Foo_(bar)", + "https://example.org/a?b=1", + "meshcore://channel/add?name=Ops&secret=ab", + ] + + +def test_a_link_without_a_scheme_is_found_by_www_or_a_delegated_tld() -> None: + """``www.`` or a real TLD says *link*; each comes back as the https URL it opens.""" + body = ( + "map at meshterm.net/map, flasher at flasher.meshcore.co.uk. " + "WWW.Example.org (or www.anything.local) and github.com/jpmartineau/MeshTerm?tab=1!" + ) + assert urls(body) == [ + "https://meshterm.net/map", + "https://flasher.meshcore.co.uk", + "https://WWW.Example.org", + "https://www.anything.local", # www. is enough on its own; .local is no TLD + "https://github.com/jpmartineau/MeshTerm?tab=1", + ] + + +def test_a_dotted_word_that_is_no_link_stays_text() -> None: + """A file, a number, an abbreviation, an address, a path: no TLD, or not a word's start.""" + body = ( + "see file.txt and notes.pdf, pi is 3.14 and v1.2.3, e.g. this, i.e. that, " + "mail jp@meshterm.net, open C:/logs/meshterm.log or ./run.sh, @Alice.Smith" + ) + assert urls(body) == [] + + +def test_the_same_link_written_two_ways_is_found_once() -> None: + """``meshterm.net`` and ``https://meshterm.net`` open the same page — one code, not two.""" + assert urls("meshterm.net or https://meshterm.net") == ["https://meshterm.net"] + # A host inside a schemed link is part of it, never a second link of its own. + assert urls("https://github.com/x/meshterm.net") == ["https://github.com/x/meshterm.net"] + + +def test_the_tld_registry_is_ianas_list_read_whole() -> None: + """The file is IANA's, version line and all; the set is every TLD in it, lower case.""" + assert TLDS_FILE.read_text(encoding="ascii").startswith("# Version ") + assert {"com", "net", "org", "dev", "io", "ca", "uk"} <= tlds() + assert not {"txt", "pdf", "log", "local"} & tlds() + assert len(tlds()) > 1000