# Gemma-4 Ayurveda Training Dataset

Comprehensive conversation dataset for fine-tuning Gemma-4 on Charaka Samhita Ayurvedic knowledge.

## Dataset Statistics

- **Total Conversations:** 51,583
- **Source Shlokas:** 3,439
- **Sections Covered:** 8 (SU, NI, VI, SHA, IND, CI/CHI, KA/KAL, SI/SID)
- **Estimated Size:** ~235 MB (JSONL), ~169 MB (CSV)
- **Languages:** English (60%), Hindi (25%), Sanskrit (15%)

## File Formats

### 1. `gemma4_ayurveda_complete.jsonl` (ShareGPT Format)

Complete dataset with all metadata and conversation types.

```json
{
  "shloka_id": "CH_SU_1/1",
  "conversation_type": "natural_conceptual_en",
  "messages": [
    {"role": "user", "content": "..."},
    {"role": "model", "content": "..."}
  ],
  "metadata": {
    "sanskrit": "अथातो दीर्घञ्जीवितीयमध्यायं व्याख्यास्यामः",
    "transliteration": "athāto dīrghañjīvitīyamadhyāyaṃ vyākhyāsyāmaḥ",
    "section": "Sutrasthana (सूत्रस्थान)",
    "section_hi": "सूत्रस्थान",
    "section_code": "SU",
    "section_description": "General Principles and Fundamentals",
    "chapter": 1,
    "chapter_name": "Dīrghañjīvitīyamadhyāyaḥ (दीर्घञ्जीवितीयाध्यायः)",
    "verse": 1,
    "translation_en": "Now (I) shall expound the chapter on longevity.",
    "translation_hi": ""
  }
}
```

### 2. `gemma4_ayurveda_training.csv` (Alpaca Format)

Simplified format for direct training:

| Column | Description |
|--------|-------------|
| instruction | User question |
| input | Empty (for compatibility) |
| output | Assistant answer |
| shloka_id | Reference ID |
| conversation_type | Question category |
| sanskrit | Original Sanskrit text |
| transliteration | IAST transliteration |

## Conversation Types

### Natural Questions (No text references)
- `natural_conceptual_en/hi/sa` - Conceptual understanding
- `natural_analytical_en` - Deep reasoning
- `natural_application_en` - Clinical application

### Reference Questions (With IDs)
- `reference_translation` - Direct translation requests
- `reference_id_lookup` - ID-based queries
- `reference_complete_details` - Full information
- `reference_chapter_context` - Location queries
- `reference_multi_turn` - Multi-turn conversations

### Advanced Questions
- `advanced_clinical_case` - Patient scenarios
- `advanced_word_analysis` - Technical term analysis
- `advanced_historical_context` - Philosophical background
- `advanced_comparative` - Cross-text comparison

### Cross-Reference Questions
- `cross_reference_intra_chapter` - Within chapter connections

## Sections Covered

| Code | Name (EN) | Name (HI) | Description |
|------|-----------|-----------|-------------|
| SU | Sutrasthana (सूत्रस्थान) | सूत्रस्थान | General Principles |
| NI | Nidanasthana (निदानस्थान) | निदानस्थान | Pathology |
| VI | Vimanasthana (विमानस्थान) | विमानस्थान | Measurements |
| SHA | Sharirasthana (शारीरस्थान) | शारीरस्थान | Anatomy |
| IND | Indriyasthana (इन्द्रियस्थान) | इन्द्रियस्थान | Sensory Organs |
| CI/CHI | Cikitsasthana (चिकित्सास्थान) | चिकित्सास्थान | Treatment |
| KA/KAL | Kalpasthana (कल्पस्थान) | कल्पस्थान | Pharmaceuticals |
| SI/SID | Siddhisthana (सिद्धिस्थान) | सिद्धिस्थान | Perfection |

## Usage

### Training with Unsloth

```python
python train_gemma4_ayurveda.py
```

### Loading Dataset

```python
import json

# ShareGPT format
with open('gemma4_ayurveda_complete.jsonl', 'r') as f:
    conversations = []
    content = f.read()
    objects = content.split('\n}\n{')
    for i, obj in enumerate(objects):
        if i == 0:
            obj = obj + '\n}'
        else:
            obj = '{\n' + obj + '\n}'
        conversations.append(json.loads(obj))

# Alpaca format
import pandas as pd
df = pd.read_csv('gemma4_ayurveda_training.csv')
```

## Training Script

See `train_gemma4_ayurveda.py` for complete training setup with:
- 4-bit quantization (QLoRA)
- LoRA rank 32
- Max sequence length 4096
- Batch size 2, gradient accumulation 4
- Learning rate 2e-4

## Dataset Generation

Generated using `generate_gemma4_dataset.py`:

```python
python generate_gemma4_dataset.py
```

## Full Chapter Names

All answers include complete Sanskrit chapter names:
- **Sutrasthana (सूत्रस्थान)**
- **Dīrghañjīvitīyamadhyāyaḥ (दीर्घञ्जीवितीयाध्यायः)**
- **Nidānasthānam (निदानस्थानम्)**
- etc.

## Quality Features

1. **No hallucination triggers:** Natural questions embed specific details
2. **Cross-references:** Link related verses within chapters
3. **Full metadata:** Preserve all Sanskrit, translations, citations
4. **Multiple formats:** ShareGPT for flexibility, Alpaca for training
5. **Ascending order:** Sorted by shloka_id (CH_SU_1/1 → CH_SI_12/60)

## License

Apache-2.0 (matches Gemma-4 license)