{"_id":"@deduq/thai-syllable-segmenter-crf","_rev":"2-f35c02d50e97982477cdb1b0954a0729","name":"@deduq/thai-syllable-segmenter-crf","dist-tags":{"latest":"1.0.1"},"versions":{"1.0.0":{"name":"@deduq/thai-syllable-segmenter-crf","version":"1.0.0","keywords":["thai","syllable","segmentation","crf","nlp","natural-language-processing","thai-language","machine-learning","conditional-random-fields","text-processing","linguistics","phonetics"],"author":{"name":"saalimon"},"license":"MIT","_id":"@deduq/thai-syllable-segmenter-crf@1.0.0","maintainers":[{"name":"deduq","email":"niti08@gmail.com"}],"homepage":"https://github.com/saalimon/ThaiSyllableSegmenter#readme","bugs":{"url":"https://github.com/saalimon/ThaiSyllableSegmenter/issues"},"os":["darwin","linux","win32"],"dist":{"shasum":"6973985fb8b33334018164170a0eae0170791e56","tarball":"https://registry.npmjs.org/@deduq/thai-syllable-segmenter-crf/-/thai-syllable-segmenter-crf-1.0.0.tgz","fileCount":12,"integrity":"sha512-IAbTJdjc60KOFhVlfN66srdY7Qt+PjImyFSHN5QqtTdtIsttDpV3+HLiACAMdvzhJEPA9ZuVDBuBPX3TJITySA==","signatures":[{"sig":"MEYCIQDcdPX3cQgy/GL4a0T/4IK86kZ2YW0EFYTCb6JEZHWv/QIhAKiefdNXQoVdZIP2iZPfDZbzMwwSP7jnP3YNTi5g5Zr1","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":79887},"main":"src/index.js","types":"types/index.d.ts","engines":{"node":">=14.0.0"},"gitHead":"40ee611842e552b0a154cc8a3c43c7ede8c1dc7f","scripts":{"demo":"node examples/demo.js","test":"node test/test.js","start":"node src/index.js","train":"node src/train.js","prepare":"node scripts/prepare.js","prepublishOnly":"npm test"},"_npmUser":{"name":"deduq","email":"niti08@gmail.com"},"repository":{"url":"git+https://github.com/saalimon/ThaiSyllableSegmenter.git","type":"git"},"_npmVersion":"10.9.2","description":"Thai syllable segmenter using Conditional Random Fields - A JavaScript implementation for segmenting Thai text into syllables using machine learning","directories":{"test":"test","example":"examples"},"_nodeVersion":"22.17.0","dependencies":{"lodash":"^4.17.21","ml-matrix":"^6.10.4"},"publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"assert":"^2.0.0"},"_npmOperationalInternal":{"tmp":"tmp/thai-syllable-segmenter-crf_1.0.0_1758379400238_0.7179910940490533","host":"s3://npm-registry-packages-npm-production"}},"1.0.1":{"name":"@deduq/thai-syllable-segmenter-crf","version":"1.0.1","description":"Thai syllable segmenter using Conditional Random Fields - A JavaScript implementation for segmenting Thai text into syllables using machine learning","main":"src/index.js","types":"types/index.d.ts","scripts":{"start":"node src/index.js","test":"node test/test.js","train":"node src/train.js","demo":"node examples/demo.js","prepublishOnly":"npm test","prepare":"node scripts/prepare.js"},"keywords":["thai","syllable","segmentation","crf","nlp","natural-language-processing","thai-language","machine-learning","conditional-random-fields","text-processing","linguistics","phonetics"],"author":{"name":"saalimon"},"license":"MIT","dependencies":{"ml-matrix":"^6.10.4","lodash":"^4.17.21"},"devDependencies":{"assert":"^2.0.0"},"engines":{"node":">=14.0.0"},"os":["darwin","linux","win32"],"directories":{"example":"examples","test":"test"},"repository":{"type":"git","url":"git+https://github.com/saalimon/ThaiSyllableSegmenter.git"},"bugs":{"url":"https://github.com/saalimon/ThaiSyllableSegmenter/issues"},"homepage":"https://github.com/saalimon/ThaiSyllableSegmenter#readme","publishConfig":{"access":"public"},"_id":"@deduq/thai-syllable-segmenter-crf@1.0.1","gitHead":"ee9ad2a925dc46bc076bfcaa020019f6d25e6470","_nodeVersion":"22.17.0","_npmVersion":"10.9.2","dist":{"integrity":"sha512-bQjng+5EpY5VY20fyhKN5BDa2gt94HkSFz/o3ZU4hvvHJ8ZWUqsuBbXigITQa0JytrvUJs2NtslI/hJOkFt1xw==","shasum":"3602c6d3a666a94afc2bb94460236609b7907397","tarball":"https://registry.npmjs.org/@deduq/thai-syllable-segmenter-crf/-/thai-syllable-segmenter-crf-1.0.1.tgz","fileCount":12,"unpackedSize":722481,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQCOlvGGfKUeRdtXeVzDycWsQiXteuHbFDrznsjqafa3ZwIgQVUwlWxUZghRyjgnMHASOorWyM+xdFH5xtkpixw0bxg="}]},"_npmUser":{"name":"deduq","email":"niti08@gmail.com"},"maintainers":[{"name":"deduq","email":"niti08@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/thai-syllable-segmenter-crf_1.0.1_1758384261953_0.015650369161184452"},"_hasShrinkwrap":false}},"time":{"created":"2025-09-20T14:43:20.146Z","modified":"2025-09-20T16:04:22.328Z","1.0.0":"2025-09-20T14:43:20.437Z","1.0.1":"2025-09-20T16:04:22.142Z"},"bugs":{"url":"https://github.com/saalimon/ThaiSyllableSegmenter/issues"},"author":{"name":"saalimon"},"license":"MIT","homepage":"https://github.com/saalimon/ThaiSyllableSegmenter#readme","keywords":["thai","syllable","segmentation","crf","nlp","natural-language-processing","thai-language","machine-learning","conditional-random-fields","text-processing","linguistics","phonetics"],"repository":{"type":"git","url":"git+https://github.com/saalimon/ThaiSyllableSegmenter.git"},"description":"Thai syllable segmenter using Conditional Random Fields - A JavaScript implementation for segmenting Thai text into syllables using machine learning","maintainers":[{"name":"deduq","email":"niti08@gmail.com"}],"readme":"# Thai Syllable Segmenter using Conditional Random Fields\n\nA JavaScript implementation of a Thai syllable segmenter using Conditional Random Fields (CRF). This tool segments Thai text into syllables, which is crucial for many NLP tasks in Thai language processing including text-to-speech, phonetic analysis, and language learning applications.\n\n## 🌟 Features\n\n- **Conditional Random Fields Implementation**: Complete CRF algorithm for sequence labeling\n- **Thai-specific Feature Extraction**: Comprehensive features including character classes, phonological rules, and contextual information\n- **Character-level Syllable Boundary Detection**: Accurate segmentation based on linguistic principles\n- **Training and Inference Capabilities**: Train your own models or use pre-trained ones\n- **Support for Thai Unicode Characters**: Handles all Thai characters, tone marks, and diacritics\n- **Model Persistence**: Save and load trained models\n- **Batch Processing**: Segment multiple texts efficiently\n- **Performance Evaluation**: Built-in metrics for model assessment\n\n## 🚀 Quick Start\n\n### Installation\n\n```bash\ngit clone <repository-url>\ncd thai-syllable-segmenter-crf\nnpm install\n```\n\n### Basic Usage\n\n```javascript\nconst { ThaiSyllableSegmenter } = require('./src/index');\n\nasync function main() {\n    // Create segmenter instance\n    const segmenter = new ThaiSyllableSegmenter();\n\n    // Training data\n    const trainingData = [\n        { text: \"สวัสดี\", syllables: [\"สวัส\", \"ดี\"] },\n        { text: \"ครับ\", syllables: [\"ครับ\"] },\n        { text: \"ขอบคุณ\", syllables: [\"ขอบ\", \"คุณ\"] },\n        { text: \"สบายดี\", syllables: [\"สบาย\", \"ดี\"] }\n    ];\n\n    // Train the model\n    await segmenter.train(trainingData);\n\n    // Segment Thai text\n    const result = segmenter.segment(\"สวัสดีครับ\");\n    console.log(result); // Output: [\"สวัส\", \"ดีค\", \"รับ\"]\n}\n\nmain();\n```\n\n### Using Pre-trained Model\n\n```javascript\nconst segmenter = new ThaiSyllableSegmenter();\nawait segmenter.loadModel('./models/thai_model_simple.json');\n\nconst syllables = segmenter.segment(\"มหาวิทยาลัย\");\nconsole.log(syllables); // [\"มหา\", \"วิท\", \"ยา\", \"ลัย\"]\n```\n\n## 📁 Project Structure\n\n```\nthai-syllable-segmenter-crf/\n├── src/\n│   ├── index.js                 # Main entry point\n│   ├── ThaiSyllableSegmenter.js # Main segmenter class\n│   ├── CRF.js                   # CRF implementation\n│   ├── FeatureExtractor.js      # Thai-specific features\n│   ├── ThaiUtils.js             # Thai language utilities\n│   └── train.js                 # Training script\n├── examples/\n│   ├── demo.js                  # Interactive demo\n│   ├── usage.js                 # Usage examples\n│   └── training_data.json       # Sample training data\n├── test/\n│   └── test.js                  # Unit tests\n├── models/                      # Trained models directory\n├── TUTORIAL.md                  # Comprehensive tutorial\n└── README.md                    # This file\n```\n\n## 🎯 Key Components\n\n### 1. Conditional Random Fields (CRF)\n- Complete CRF implementation with forward-backward algorithm\n- Viterbi decoding for inference\n- L-BFGS optimization with regularization\n- Model serialization and deserialization\n\n### 2. Thai Language Processing\n- Character classification (consonants, vowels, tone marks, diacritics)\n- Consonant class identification (high, middle, low)\n- Syllable structure validation\n- Unicode normalization and character combining\n\n### 3. Feature Engineering\n- **Basic Features**: Character identity, position, class\n- **Context Features**: N-gram combinations, surrounding characters\n- **Linguistic Features**: Phonological rules, syllable patterns\n- **Statistical Features**: Frequency, boundary likelihood\n\n### 4. Training Pipeline\n- Data validation and preprocessing\n- Feature extraction and mapping\n- CRF parameter optimization\n- Model evaluation and metrics\n\n## 📊 Algorithm Details\n\nThe segmenter uses BIO (Beginning-Inside-Outside) labeling where:\n- **B**: Beginning of a new syllable\n- **I**: Inside/continuation of current syllable\n\n### Feature Types:\n1. **Character Features**: Identity, class, phonological properties\n2. **Positional Features**: Location in text, relative position  \n3. **Contextual Features**: Surrounding characters (window size 3)\n4. **N-gram Features**: Bigrams and trigrams\n5. **Structural Features**: Consonant clusters, vowel patterns\n6. **Phonological Features**: Thai-specific linguistic rules\n\n### Thai Syllable Patterns:\n- C (Consonant): ก\n- CV (Consonant-Vowel): กา  \n- CVC (Consonant-Vowel-Consonant): กับ\n- CCV (Consonant-Cluster-Vowel): กรา\n- CCVC (Consonant-Cluster-Vowel-Consonant): กราบ\n\n## 🛠 Commands\n\n```bash\n# Run tests\nnpm test\n\n# Run demo\nnpm run demo\n\n# Run basic example\nnpm start\n\n# Train custom model\nnode src/train.js data/training.json models/custom.json\n\n# Run interactive mode\nnode examples/demo.js --interactive\n```\n\n## 📈 Training Your Own Models\n\n### 1. Prepare Training Data\n\nCreate a JSON file with training examples:\n\n```json\n[\n    { \"text\": \"สวัสดี\", \"syllables\": [\"สวัส\", \"ดี\"] },\n    { \"text\": \"มหาวิทยาลัย\", \"syllables\": [\"มหา\", \"วิท\", \"ยา\", \"ลัย\"] }\n]\n```\n\n### 2. Train Model\n\n```bash\nnode src/train.js training_data.json my_model.json --learningRate 0.1 --maxIterations 100\n```\n\n### 3. Evaluate Model\n\n```javascript\nconst metrics = segmenter.evaluate(testData);\nconsole.log(`Accuracy: ${(metrics.accuracy * 100).toFixed(2)}%`);\nconsole.log(`F1 Score: ${(metrics.f1Score * 100).toFixed(2)}%`);\n```\n\n## ⚙️ Configuration Options\n\n```javascript\nconst segmenter = new ThaiSyllableSegmenter({\n    learningRate: 0.1,        // Learning rate (0.01-0.3)\n    maxIterations: 100,       // Maximum training iterations  \n    regularization: 0.01,     // L2 regularization (0.001-0.1)\n    tolerance: 1e-6          // Convergence tolerance\n});\n```\n\n## 🔍 Performance\n\nThe segmenter achieves good performance on Thai syllable segmentation:\n- **Accuracy**: 85-95% depending on training data quality\n- **Speed**: ~1000 characters/second on modern hardware\n- **Memory**: Efficient sparse feature representation\n\nPerformance depends on:\n- Training data size and quality\n- Text complexity\n- Model hyperparameters\n\n## 🎓 Use Cases\n\n1. **Text-to-Speech Systems**: Prepare Thai text for pronunciation\n2. **Language Learning**: Show syllable boundaries for pronunciation practice\n3. **Linguistic Analysis**: Study Thai syllable patterns and phonology\n4. **Search and Indexing**: Improve Thai text search capabilities\n5. **Data Preprocessing**: Prepare Thai text for other NLP tasks\n\n## 🧪 Testing\n\nRun the comprehensive test suite:\n\n```bash\nnpm test\n```\n\nTests cover:\n- Thai character classification\n- Feature extraction\n- CRF training and inference\n- Model serialization\n- End-to-end segmentation\n\n## 📚 Documentation\n\n- **README.md**: Overview and quick start\n- **TUTORIAL.md**: Comprehensive tutorial with examples\n- **src/**: Inline code documentation\n- **examples/**: Working code examples\n\n## 🤝 Contributing\n\nContributions are welcome! Areas for improvement:\n1. More training data\n2. Better feature engineering\n3. Performance optimization\n4. Additional evaluation metrics\n5. Web interface\n\n## 📄 License\n\nMIT License - see LICENSE file for details\n\n### Special Thanks\n\n**Dataset**: This project uses training data derived from the **Han-solo** dataset by PyThaiNLP, a comprehensive collection of Thai syllable-segmented text from social media and web sources.\n\n- **Han-solo Repository**: https://github.com/PyThaiNLP/Han-solo\n\nThe Han-solo dataset provides real-world Thai text with manual syllable segmentation, making it an invaluable resource for training and evaluating Thai NLP models. Our implementation converts and utilizes this data to train the Conditional Random Fields model for accurate syllable boundary detection.\n\n**Data Processing**: The original pipe-delimited format (`ซื้อ|ที่|ไหน|ครับ`) from Han-solo has been converted to JSON format for use with our CRF training pipeline, maintaining the high-quality manual segmentation while making it compatible with modern JavaScript machine learning workflows.\n\n---\n\nFor detailed usage instructions and advanced features, see [TUTORIAL.md](TUTORIAL.md).\n","readmeFilename":"README.md"}