Detalhes do pacote

@tsports/uniseg

tsports12MIT0.4.7-tsport

Complete TypeScript port of rivo/uniseg with 100% API compatibility. Unicode text segmentation for grapheme clusters, word boundaries, and text width calculation.

uniseg, unicode, text, segmentation

readme (leia-me)

Uniseg TypeScript

npm version TypeScript Tests codecov License: MIT Node.js Version A comprehensive TypeScript port of rivo/uniseg with 100% API compatibility Unicode text segmentation for grapheme clusters, text width calculation, and string manipulation. Built for TypeScript/Node.js. Documentation • Examples • API Reference • Go Original

✨ Features

  • 🔤 Complete Unicode Text Segmentation - Grapheme clusters, word boundaries, line breaks
  • 🌍 Unicode 15.0.0 Support - Latest Unicode standard with comprehensive property tables
  • 😀 Advanced Emoji Support - Flags, ZWJ sequences, skin tone modifiers, regional indicators
  • 🕉️ Complex Script Support - Devanagari, Bengali, and other Indic scripts with combining marks
  • 📏 Accurate Width Calculation - East Asian Width property for monospace fonts
  • 🔄 100% Go API Compatibility - Perfect compatibility with original Go rivo/uniseg
  • ⚡ High Performance - Optimized state machines and Unicode property lookups
  • 🎯 Type-Safe - Full TypeScript support with comprehensive type definitions
  • 📦 Zero Dependencies - Lightweight and self-contained
  • 🚀 Cross-Platform - Works on Windows, macOS, and Linux

🚀 Quick Start

Installation

# npm
npm install @tsports/uniseg

# yarn
yarn add @tsports/uniseg

# bun (recommended)
bun add @tsports/uniseg

Basic Usage

TypeScript-Native API (Recommended):

import { graphemeClusterCount, stringWidth, reverseString, newGraphemes } from '@tsports/uniseg';

// Count user-perceived characters (grapheme clusters)
graphemeClusterCount('Hello'); // 5
graphemeClusterCount('🇩🇪🏳️‍🌈'); // 2 (German flag + rainbow flag)
graphemeClusterCount('नमस्ते'); // 4 (Devanagari script)
graphemeClusterCount('🧑‍💻'); // 1 (person technologist emoji)

// Calculate display width for monospace fonts
stringWidth('Hello'); // 5
stringWidth('你好'); // 4 (full-width characters)
stringWidth('🇩🇪🏳️‍🌈'); // 4 (emoji width)

// Reverse strings while preserving grapheme clusters
reverseString('Hello'); // 'olleH'
reverseString('🇩🇪🏳️‍🌈'); // '🏳️‍🌈🇩🇪'
reverseString('नमस्ते'); // 'तेस्मन'

// Iterate through grapheme clusters
const iter = newGraphemes('🧑‍💻 Hello');
let cluster;
while ((cluster = iter.next()) !== null) {
  console.log(`Cluster: "${cluster.cluster}" at position ${cluster.startPos}`);
}
// Output:
// Cluster: "🧑‍💻" at position 0
// Cluster: " " at position 5
// Cluster: "H" at position 6
// Cluster: "e" at position 7
// ...

Go-Compatible API (For Go Developers):

import { GraphemeClusterCount, StringWidth, ReverseString, NewGraphemes } from '@tsports/uniseg/go-style';

// Exact Go rivo/uniseg API with PascalCase methods
const count = GraphemeClusterCount('🇩🇪🏳️‍🌈'); // 2
const width = StringWidth('Hello 世界'); // 9
const reversed = ReverseString('नमस्ते'); // 'तेस्मन'

// Go-style iterator
const iter = NewGraphemes('🧑‍💻');
while (iter.Next()) {
  const cluster = iter.Str();
  const runes = iter.Runes();
  console.log(`"${cluster}" -> [${runes.map(r => `U+${r.toString(16).toUpperCase()}`).join(', ')}]`);
}

📊 Perfect Compatibility Results

Our implementation achieves 100% compatibility with Go rivo/uniseg:

Test Case Expected Got Status
"Hello" 5 5 ✅
"🇩🇪🏳️‍🌈" (flags) 2 2 ✅
"नमस्ते" (Devanagari) 4 4 ✅
"🧑‍💻" (ZWJ emoji) 1 1 ✅
"a̧" (combining) 1 1 ✅
"" (empty) 0 0 ✅

All test outputs match the Go reference implementation exactly.

📖 Documentation

Core Functions

Grapheme Cluster Counting

import { graphemeClusterCount } from '@tsports/uniseg';

// Basic counting
graphemeClusterCount('Hello World'); // 11

// Complex emoji sequences
graphemeClusterCount('👨‍👩‍👧‍👦'); // 1 (family emoji)
graphemeClusterCount('🏳️‍⚧️'); // 1 (transgender flag)
graphemeClusterCount('👋🏻'); // 1 (waving hand with skin tone)

// Regional indicator pairs (flags)
graphemeClusterCount('🇺🇸🇬🇧'); // 2 (US flag + UK flag)

// Complex scripts with combining marks
graphemeClusterCount('ज़िन्दगी'); // 7 (Hindi with nukta and combining marks)
graphemeClusterCount('பெண்கள்'); // 6 (Tamil script)

String Width Calculation

import { stringWidth } from '@tsports/uniseg';

// Latin characters
stringWidth('Hello'); // 5

// East Asian characters (full-width)
stringWidth('你好世界'); // 8
stringWidth('こんにちは'); // 10

// Mixed content
stringWidth('Hello 世界'); // 9

// Emoji and symbols
stringWidth('🚀📱💻'); // 6
stringWidth('→←↑↓'); // 4

String Reversal

import { reverseString } from '@tsports/uniseg';

// Preserves grapheme cluster integrity
reverseString('Café'); // 'éfaC'
reverseString('🇺🇸🇬🇧'); // '🇬🇧🇺🇸'
reverseString('👨‍👩‍👧‍👦 Family'); // 'ylimaF 👨‍👩‍👧‍👦'

// Complex scripts
reverseString('नमस्ते दुनिया'); // 'ायिनुद ेत्समन'
reverseString('السلام عليكم'); // 'مكيلع مالسلا'

Advanced Iteration

import { newGraphemes, stepString } from '@tsports/uniseg';

// Iterator pattern
const iter = newGraphemes('🧑‍💻 Hello');
let cluster;
while ((cluster = iter.next()) !== null) {
  console.log({
    cluster: cluster.cluster,
    runes: cluster.runes,
    position: cluster.startPos,
    length: cluster.length
  });
}

// Step-by-step processing
let str = '🇩🇪🏳️‍🌈';
let state = -1;
while (str.length > 0) {
  const result = stepString(str, state);
  console.log(`Segment: "${result.segment}"`);
  str = result.remainder;
  state = result.newState;
}

Unicode Standards Compliance

This library implements:

  • UAX #29 - Unicode Text Segmentation
  • UAX #11 - East Asian Width
  • UAX #15 - Unicode Normalization Forms
  • Unicode 15.0.0 - Latest Unicode standard

Supported Features

Emoji Sequences

  • Regional Indicator Sequences: 🇺🇸 🇬🇧 🇯🇵
  • ZWJ Sequences: 👨‍💻 👩‍🔬 🏳️‍🌈 🏳️‍⚧️
  • Modifier Sequences: 👋🏻 👋🏿 💪🏽
  • Flag Sequences: 🏴󠁧󠁢󠁳󠁣󠁴󠁿 (Scotland)
  • Keycap Sequences: 1️⃣ 2️⃣ #️⃣ *️⃣

Complex Scripts

  • Indic Scripts: Devanagari, Bengali, Tamil, Telugu, etc.
  • Arabic Script: Arabic, Persian, Urdu with joining behavior
  • Combining Marks: Diacritics, accents, nukta marks
  • Hangul: Korean syllable composition

Text Width

  • East Asian Width properties (Narrow, Wide, Fullwidth, Halfwidth, Ambiguous)
  • Emoji width calculation with presentation selectors
  • Combining mark handling (zero-width)
  • Control character handling

🔄 Dual API Support

TypeScript-Native API (Recommended)

Modern TypeScript patterns with camelCase methods:

import { graphemeClusterCount, stringWidth, newGraphemes } from '@tsports/uniseg';

const count = graphemeClusterCount('🇩🇪🏳️‍🌈');
const width = stringWidth('Hello 世界');
const iter = newGraphemes('text');

Go-Compatible API

100% identical Go rivo/uniseg API with PascalCase methods:

import { GraphemeClusterCount, StringWidth, NewGraphemes } from '@tsports/uniseg/go-style';

// Exact Go API
const count = GraphemeClusterCount('🇩🇪🏳️‍🌈');
const width = StringWidth('Hello 世界');
const iter = NewGraphemes('text');

Go → TypeScript Migration

// Go rivo/uniseg
import "github.com/rivo/uniseg"

count := uniseg.GraphemeClusterCount("🇩🇪🏳️‍🌈")
width := uniseg.StringWidth("Hello 世界")
// TypeScript - EXACT same API
import { GraphemeClusterCount, StringWidth } from '@tsports/uniseg/go-style';

const count = GraphemeClusterCount('🇩🇪🏳️‍🌈');
const width = StringWidth('Hello 世界');

🧪 Testing & Quality Assurance

100% API Compatibility Verified

  • Comprehensive Testing - All outputs compared with Go reference implementation
  • Unicode Compliance - Full Unicode 15.0.0 test suite coverage
  • Cross-Platform Testing - Windows, macOS, Linux validation
  • Performance Testing - Benchmarked against Go implementation
  • Edge Case Coverage - Complex sequences, boundary conditions, error cases

Test Execution

# Run all tests
bun test

# Run Go compatibility tests
bun test test/automated-cases.test.ts

# Test specific functionality
cd test/corpus/basic/001-grapheme-count
go run case.go        # Expected output
bun --bun run case.ts # Our output
diff <(go run case.go) <(bun --bun run case.ts) # Should be identical

⚡ Performance

Optimized for high performance with:

  • Fast Unicode property lookups - Binary search on sorted ranges
  • Efficient state machines - Minimal memory allocations
  • Optimized algorithms - Based on proven Go implementation
  • TypeScript compilation - Full ahead-of-time optimization

Run benchmarks:

bun run benchmark

📋 Examples

Emoji Analysis

import { graphemeClusterCount, stringWidth, newGraphemes } from '@tsports/uniseg';

const emojis = [
  '👋',           // Basic emoji
  '👋🏻',          // Emoji + skin tone modifier
  '👨‍💻',          // ZWJ sequence (man technologist)
  '👨‍👩‍👧‍👦',       // Family ZWJ sequence
  '🏳️‍🌈',         // Rainbow flag
  '🏳️‍⚧️',         // Transgender flag
  '🇺🇸',          // Country flag
  '1️⃣',          // Keycap sequence
  '🏴󠁧󠁢󠁳󠁣󠁴󠁿'       // Subdivision flag (Scotland)
];

emojis.forEach(emoji => {
  console.log({
    emoji,
    clusters: graphemeClusterCount(emoji),
    width: stringWidth(emoji),
    codePoints: [...emoji].map(c => `U+${c.codePointAt(0)?.toString(16).toUpperCase()}`)
  });
});

Text Processing

import { graphemeClusterCount, reverseString, newGraphemes } from '@tsports/uniseg';

function analyzeText(text: string) {
  const clusters = [];
  const iter = newGraphemes(text);
  let cluster;

  while ((cluster = iter.next()) !== null) {
    clusters.push({
      text: cluster.cluster,
      position: cluster.startPos,
      codePoints: cluster.runes.length
    });
  }

  return {
    originalText: text,
    reversedText: reverseString(text),
    totalClusters: graphemeClusterCount(text),
    displayWidth: stringWidth(text),
    clusters
  };
}

// Analyze complex text
const result = analyzeText('Hello 🌍! नमस्ते 🇮🇳');
console.log(JSON.stringify(result, null, 2));

Go API Compatibility Demo

import {
  GraphemeClusterCount,
  StringWidth,
  ReverseString,
  NewGraphemes
} from '@tsports/uniseg/go-style';

// Direct Go API usage
const text = '🇩🇪🏳️‍🌈 Hello, 世界!';

console.log('Go-Compatible API Results:');
console.log(`Text: "${text}"`);
console.log(`Grapheme Clusters: ${GraphemeClusterCount(text)}`);
console.log(`Display Width: ${StringWidth(text)}`);
console.log(`Reversed: "${ReverseString(text)}"`);

// Iterator usage (Go-style)
const iter = NewGraphemes(text);
console.log('\nGrapheme Clusters:');
while (iter.Next()) {
  const cluster = iter.Str();
  const runes = iter.Runes();
  console.log(`  "${cluster}" [${runes.map(r => `U+${r.toString(16).toUpperCase()}`).join(', ')}]`);
}

🏗️ Architecture

Project Structure

src/
├── index.ts              # TypeScript-native API exports
├── go-style.ts           # Go-compatible API wrapper
├── core.ts               # Core grapheme cluster functions
├── properties.ts         # Unicode 15.0.0 property tables
├── grapheme-rules.ts     # UAX #29 state machine implementation
├── step.ts               # Combined boundary detection
├── width.ts              # East Asian Width calculation
├── types.ts              # TypeScript type definitions
└── utils.ts              # Utility functions

Design Principles

  1. 100% Go Compatibility - Identical behavior and API
  2. Performance First - Optimized algorithms and data structures
  3. Type Safety - Comprehensive TypeScript types
  4. Unicode Standards - Full compliance with Unicode specifications
  5. Zero Dependencies - Self-contained implementation

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

# Clone with submodules (includes Go reference)
git clone --recursive https://github.com/tsports/uniseg.git
cd uniseg

# Install dependencies
bun install

# Build and test
bun run build
bun test

# Test Go compatibility
cd test/corpus/basic/001-grapheme-count
diff <(go run case.go) <(bun --bun run case.ts)

Adding Unicode Test Cases

When contributing Unicode-related features:

  1. Test with Go first to get expected behavior
  2. Include complex examples with edge cases
  3. Add comprehensive test coverage
  4. Verify Unicode standard compliance

📊 Browser Support

While designed for Node.js environments, the core algorithms work in modern browsers:

  • ES2020+ - Uses modern JavaScript features
  • Unicode support - Requires JavaScript Unicode support
  • TypeScript - Full type support in development

🔗 Links

📄 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

This TypeScript port is made possible by the exceptional work of the original Go library creators:

  • Oliver Kuederle - Creator of the excellent original rivo/uniseg library that serves as the foundation for this TypeScript implementation
  • Unicode Consortium - For maintaining the Unicode standard and comprehensive specifications
  • Go Team - For the inspiring programming language and well-designed standard library
  • TypeScript Community - For the excellent tooling and ecosystem that makes this port possible

This is a TypeScript port - All credit for the original design, algorithms, and Unicode expertise goes to the rivo/uniseg project and its contributors.

Made with ❤️ by Saulo Vallory GitHub
Bringing Unicode text processing excellence to the TypeScript ecosystem

Built on the foundation of rivo/uniseg by Oliver Kuederle

changelog (log de mudanças)

Changelog

All notable changes to @tsports/uniseg will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

  • Initial TypeScript port of rivo/uniseg v0.4.7
  • Complete Unicode 15.0.0 property tables
  • 100% Go API compatibility with dual API support
  • TypeScript-native API with modern patterns
  • Go-compatible API with identical naming
  • Comprehensive grapheme cluster boundary detection
  • Support for complex emoji sequences (flags, ZWJ sequences, modifiers)
  • Support for Indic scripts (Devanagari, Bengali, etc.)
  • Character width calculation for monospace fonts
  • String reversal preserving grapheme clusters
  • Iterator pattern for grapheme cluster traversal
  • Extensive test suite with Go compatibility verification

Features

  • Grapheme Cluster Segmentation - Full UAX #29 implementation
  • Emoji Support - Regional indicators, ZWJ sequences, skin tone modifiers
  • Script Support - Devanagari, Bengali, and other complex scripts
  • Width Calculation - East Asian Width property support
  • Dual APIs - TypeScript-native and Go-compatible interfaces

Technical

  • TypeScript with strict type checking
  • ESM modules with proper exports
  • Zero runtime dependencies
  • Cross-platform compatibility (Windows, macOS, Linux)
  • Bun and Node.js support
  • Comprehensive CI/CD pipeline
  • Automated Go compatibility testing

[1.0.0] - TBD

Initial release - complete TypeScript port of rivo/uniseg with 100% API compatibility.