Testing Requirements

Every encoding in Mistaber must be accompanied by comprehensive tests. Tests ensure halachic accuracy, verify correct inheritance behavior, and prevent regressions. This document specifies minimum test coverage, test file organization, and strategies for writing effective tests.

Minimum Test Coverage

Required Test Counts

Test Type	Minimum	Purpose
Positive cases	5	Verify rule fires correctly
Negative cases	3	Verify rule does not fire incorrectly
Edge cases	2	Test boundary conditions
Multi-world	If applicable	Verify inheritance and overrides

Coverage by Rule Type

Rule Type	Additional Requirements
D'oraita	Extra scrutiny, 3+ edge cases
Machloket	Tests for BOTH positions
Override	Test parent and child behavior
Helper predicate	Test in isolation and integration

Test File Organization

Directory Structure

tests/
  corpus/
    yd_87/
      test_beheima_chalav.py      # YD 87:1 tests
      test_of_chaya_chalav.py     # YD 87:3 (of/chaya) tests
      test_fish_dairy.py          # YD 87:3 (fish) tests
      conftest.py                 # Shared fixtures
    yd_89/
      test_waiting_time.py        # YD 89 tests
      conftest.py
  worlds/
    test_inheritance.py           # World inheritance tests
    test_override.py              # Override mechanism tests
    test_machloket.py             # Machloket detection tests
  integration/
    test_full_queries.py          # End-to-end tests

File Naming Convention

Pattern	Example	Content
test_[topic].py	test_beheima_chalav.py	Tests for specific topic
test_[siman]_[seif].py	test_87_1.py	Tests for specific seif
conftest.py	conftest.py	Shared fixtures

Test Class Structure

import pytest
from mistaber import query

class TestBeheimaChalaviIssur:
    """Tests for beheima + chalav d'oraita prohibition (YD 87:1)."""

    @pytest.fixture
    def beheima_chalav_mixture(self):
        """Standard beheima + chalav mixture for testing."""
        return """
            mixture(m1).
            food(beef). food_type(beef, beheima).
            food(milk). food_type(milk, chalav).
            contains(m1, beef). contains(m1, milk).
        """

    # === Positive Tests (5 minimum) ===

    def test_issur_achiila_detected(self, beheima_chalav_mixture):
        """Eating prohibition is detected for beheima+chalav."""
        result = query(beheima_chalav_mixture, world="mechaber")
        assert result.holds("issur(achiila, m1, d_oraita)")

    # ... more positive tests

    # === Negative Tests (3 minimum) ===

    def test_parve_no_issur(self):
        """Parve foods do not trigger basar bechalav."""
        # ...

    # === Edge Cases (2 minimum) ===

    def test_single_food_not_mixture(self):
        """Single food item is not a mixture."""
        # ...

Writing Effective Test Cases

Positive Test Pattern

Test that rules fire when they should:

def test_rule_fires_correctly(self):
    """[Description of what the rule should do]."""
    # Arrange: Set up the scenario
    setup = """
        mixture(m1).
        food(beef). food_type(beef, beheima).
        food(milk). food_type(milk, chalav).
        contains(m1, beef). contains(m1, milk).
    """

    # Act: Run the query
    result = query(setup, world="mechaber")

    # Assert: Verify expected outcome
    assert result.holds("issur(achiila, m1, d_oraita)")
    assert result.holds("issur(bishul, m1, d_oraita)")
    assert result.holds("issur(hanaah, m1, d_oraita)")

Negative Test Pattern

Test that rules do NOT fire when they shouldn't:

def test_rule_does_not_fire_incorrectly(self):
    """[Description of scenario where rule should NOT fire]."""
    # Arrange: Set up a scenario that should NOT trigger the rule
    setup = """
        mixture(m1).
        food(carrot). food_type(carrot, parve).
        food(rice). food_type(rice, parve).
        contains(m1, carrot). contains(m1, rice).
    """

    # Act
    result = query(setup, world="mechaber")

    # Assert: Rule should NOT fire
    assert not result.holds("issur(achiila, m1, _)")
    assert not result.holds("is_beheima_chalav_mixture(m1)")

Edge Case Pattern

Test boundary conditions and special scenarios:

def test_edge_case_boundary_condition(self):
    """[Description of boundary condition being tested]."""
    # Arrange: Set up edge case scenario
    setup = """
        % Edge case: Food declared but no mixture
        food(beef). food_type(beef, beheima).
        food(milk). food_type(milk, chalav).
        % NO mixture, NO contains - just foods existing
    """

    # Act
    result = query(setup, world="mechaber")

    # Assert: No mixture-based rules should fire
    assert not result.holds("is_beheima_chalav_mixture(_)")
    assert not result.holds("issur(achiila, _, _)")

Multi-World Testing

Testing Inheritance

Verify rules propagate correctly through world hierarchy:

class TestWorldInheritance:
    """Tests for rule inheritance across worlds."""

    @pytest.fixture
    def beheima_chalav_mixture(self):
        return """
            mixture(m1).
            food(beef). food_type(beef, beheima).
            food(milk). food_type(milk, chalav).
            contains(m1, beef). contains(m1, milk).
        """

    def test_sefardi_yo_inherits_from_mechaber(self, beheima_chalav_mixture):
        """sefardi_yo should inherit mechaber's rulings."""
        mechaber_result = query(beheima_chalav_mixture, world="mechaber")
        sefardi_result = query(beheima_chalav_mixture, world="sefardi_yo")

        # Both should have same issur
        assert mechaber_result.holds("issur(achiila, m1, d_oraita)")
        assert sefardi_result.holds("issur(achiila, m1, d_oraita)")

    def test_ashk_mb_inherits_from_rema(self, beheima_chalav_mixture):
        """ashk_mb should inherit rema's rulings."""
        rema_result = query(beheima_chalav_mixture, world="rema")
        mb_result = query(beheima_chalav_mixture, world="ashk_mb")

        # Both should have same issur for beheima+chalav
        assert rema_result.holds("issur(achiila, m1, d_oraita)")
        assert mb_result.holds("issur(achiila, m1, d_oraita)")

Testing Overrides

Verify overrides prevent inheritance:

class TestOverrideBehavior:
    """Tests for override mechanism in machloket."""

    @pytest.fixture
    def fish_dairy_mixture(self):
        return """
            mixture(m1).
            food(salmon). food_type(salmon, dag).
            food(cream). food_type(cream, chalav).
            contains(m1, salmon). contains(m1, cream).
        """

    def test_mechaber_asserts_sakana(self, fish_dairy_mixture):
        """Mechaber holds fish+dairy is sakana."""
        result = query(fish_dairy_mixture, world="mechaber")
        assert result.holds("sakana(m1)")
        assert result.holds("issur(achiila, m1, sakana)")

    def test_rema_overrides_sakana(self, fish_dairy_mixture):
        """Rema overrides and permits fish+dairy."""
        result = query(fish_dairy_mixture, world="rema")
        assert result.holds("no_sakana(m1)")
        assert result.holds("heter(achiila, m1)")
        assert not result.holds("sakana(m1)")

    def test_ashk_mb_inherits_rema_override(self, fish_dairy_mixture):
        """ashk_mb should inherit rema's permission."""
        result = query(fish_dairy_mixture, world="ashk_mb")
        assert result.holds("heter(achiila, m1)")
        assert not result.holds("sakana(m1)")

    def test_sefardi_yo_inherits_mechaber_sakana(self, fish_dairy_mixture):
        """sefardi_yo should inherit mechaber's sakana."""
        result = query(fish_dairy_mixture, world="sefardi_yo")
        assert result.holds("sakana(m1)")

Testing All Seven Worlds

For critical rules, test all worlds:

@pytest.mark.parametrize("world,expected_sakana", [
    ("base", False),          # base has no fish+dairy rule
    ("mechaber", True),       # mechaber: sakana
    ("rema", False),          # rema: no sakana (override)
    ("gra", False),           # gra: likely follows rema (check)
    ("sefardi_yo", True),     # inherits from mechaber
    ("ashk_mb", False),       # inherits from rema
    ("ashk_ah", False),       # inherits from rema
])
def test_fish_dairy_across_worlds(fish_dairy_mixture, world, expected_sakana):
    """Fish+dairy sakana status varies by world."""
    result = query(fish_dairy_mixture, world=world)

    if expected_sakana:
        assert result.holds("sakana(m1)")
    else:
        assert not result.holds("sakana(m1)") or result.holds("no_sakana(m1)")

Testing Machloket

Both Positions Required

Every machloket must have tests for BOTH sides:

class TestFishDairyMachloket:
    """Tests for fish+dairy machloket between Mechaber and Rema."""

    @pytest.fixture
    def fish_dairy_mixture(self):
        return """
            mixture(m1).
            food(salmon). food_type(salmon, dag).
            food(cream). food_type(cream, chalav).
            contains(m1, salmon). contains(m1, cream).
        """

    # === Mechaber Position (5+ tests) ===

    def test_mechaber_sakana_asserted(self, fish_dairy_mixture):
        """Mechaber asserts sakana for fish+dairy."""
        result = query(fish_dairy_mixture, world="mechaber")
        assert result.holds("sakana(m1)")

    def test_mechaber_issur_achiila(self, fish_dairy_mixture):
        """Mechaber forbids eating fish+dairy."""
        result = query(fish_dairy_mixture, world="mechaber")
        assert result.holds("issur(achiila, m1, sakana)")

    def test_mechaber_position_inherited_by_sefardi_yo(self, fish_dairy_mixture):
        """Sefardi_yo inherits Mechaber's position."""
        result = query(fish_dairy_mixture, world="sefardi_yo")
        assert result.holds("sakana(m1)")

    # === Rema Position (5+ tests) ===

    def test_rema_no_sakana(self, fish_dairy_mixture):
        """Rema holds no sakana for fish+dairy."""
        result = query(fish_dairy_mixture, world="rema")
        assert result.holds("no_sakana(m1)")

    def test_rema_permits_achiila(self, fish_dairy_mixture):
        """Rema permits eating fish+dairy."""
        result = query(fish_dairy_mixture, world="rema")
        assert result.holds("heter(achiila, m1)")

    def test_rema_position_inherited_by_ashk_mb(self, fish_dairy_mixture):
        """ashk_mb inherits Rema's position."""
        result = query(fish_dairy_mixture, world="ashk_mb")
        assert result.holds("heter(achiila, m1)")

    # === Machloket Detection ===

    def test_machloket_detected(self, fish_dairy_mixture):
        """The machloket is properly marked."""
        result = query(fish_dairy_mixture, world="base")
        assert result.holds("machloket(dag_chalav_sakana, mechaber, rema, m1)")

CI Integration

Test Commands

# Run all tests
pytest tests/

# Run tests for specific siman
pytest tests/corpus/yd_87/

# Run with coverage
pytest tests/ --cov=mistaber --cov-report=html

# Run only multi-world tests
pytest tests/ -m multiworld

GitHub Actions Integration

# .github/workflows/test.yml
name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install -e ".[dev]"

      - name: Run tests
        run: pytest tests/ --cov=mistaber

      - name: Check coverage threshold
        run: |
          coverage report --fail-under=80

Pre-Commit Hook

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: pytest-check
        name: pytest
        entry: pytest tests/ -x -q
        language: system
        pass_filenames: false
        always_run: true

Test Fixtures

Shared Fixtures in conftest.py

# tests/corpus/yd_87/conftest.py
import pytest

@pytest.fixture
def beheima_chalav_mixture():
    """Standard beheima + chalav mixture."""
    return """
        mixture(m1).
        food(beef). food_type(beef, beheima).
        food(milk). food_type(milk, chalav).
        contains(m1, beef). contains(m1, milk).
    """

@pytest.fixture
def of_chalav_mixture():
    """Standard of (poultry) + chalav mixture."""
    return """
        mixture(m1).
        food(chicken). food_type(chicken, of).
        food(milk). food_type(milk, chalav).
        contains(m1, chicken). contains(m1, milk).
    """

@pytest.fixture
def chaya_chalav_mixture():
    """Standard chaya (wild animal) + chalav mixture."""
    return """
        mixture(m1).
        food(deer). food_type(deer, chaya).
        food(milk). food_type(milk, chalav).
        contains(m1, deer). contains(m1, milk).
    """

@pytest.fixture
def dag_chalav_mixture():
    """Standard dag (fish) + chalav mixture."""
    return """
        mixture(m1).
        food(salmon). food_type(salmon, dag).
        food(cream). food_type(cream, chalav).
        contains(m1, salmon). contains(m1, cream).
    """

@pytest.fixture
def parve_mixture():
    """Parve-only mixture (no basar bechalav)."""
    return """
        mixture(m1).
        food(carrot). food_type(carrot, parve).
        food(rice). food_type(rice, parve).
        contains(m1, carrot). contains(m1, rice).
    """

Parameterized Fixtures

@pytest.fixture(params=[
    ("mechaber", True),
    ("rema", True),
    ("sefardi_yo", True),
    ("ashk_mb", True),
    ("ashk_ah", True),
])
def all_worlds_beheima_expected_issur(request, beheima_chalav_mixture):
    """Parameterized fixture for testing across all worlds."""
    world, expected_issur = request.param
    return (beheima_chalav_mixture, world, expected_issur)

Test Markers

Custom Markers

# pytest.ini
[pytest]
markers =
    multiworld: tests that verify behavior across multiple worlds
    machloket: tests for disputed rulings
    doraita: tests for Torah-level prohibitions
    drabanan: tests for rabbinic prohibitions
    slow: tests that take longer to run

Using Markers

@pytest.mark.multiworld
def test_inheritance_across_worlds():
    """Test that requires checking multiple worlds."""
    pass

@pytest.mark.machloket
def test_fish_dairy_dispute():
    """Test for machloket between Mechaber and Rema."""
    pass

@pytest.mark.doraita
def test_beheima_chalav_prohibition():
    """Test for d'oraita level prohibition."""
    pass

Test Checklist

Before submitting:

• [ ] At least 5 positive test cases
• [ ] At least 3 negative test cases
• [ ] At least 2 edge cases
• [ ] Multi-world tests (if applicable)
• [ ] Machloket tests for both positions (if applicable)
• [ ] All tests pass locally
• [ ] Tests organized in correct directory
• [ ] Fixtures are shared appropriately
• [ ] Test names are descriptive

Common Testing Mistakes

Mistake 1: Insufficient Negative Tests

# WRONG - only testing positive cases
def test_issur_exists():
    result = query(beheima_chalav, world="mechaber")
    assert result.holds("issur(achiila, m1, d_oraita)")
# What about cases that should NOT have issur?

# CORRECT - include negative tests
def test_parve_has_no_issur():
    result = query(parve_only, world="mechaber")
    assert not result.holds("issur(achiila, m1, _)")

Mistake 2: Not Testing All Affected Worlds

# WRONG - only testing mechaber
def test_machloket():
    result = query(fish_dairy, world="mechaber")
    assert result.holds("sakana(m1)")
# What about rema and children?

# CORRECT - test both sides
def test_machloket_mechaber():
    result = query(fish_dairy, world="mechaber")
    assert result.holds("sakana(m1)")

def test_machloket_rema():
    result = query(fish_dairy, world="rema")
    assert not result.holds("sakana(m1)")

Mistake 3: Hardcoding Instead of Fixtures

# WRONG - repeated setup
def test_one():
    setup = """mixture(m1)..."""
    result = query(setup, world="mechaber")

def test_two():
    setup = """mixture(m1)..."""  # Duplicated!
    result = query(setup, world="rema")

# CORRECT - use fixtures
@pytest.fixture
def fish_dairy_mixture():
    return """mixture(m1)..."""

def test_one(fish_dairy_mixture):
    result = query(fish_dairy_mixture, world="mechaber")

def test_two(fish_dairy_mixture):
    result = query(fish_dairy_mixture, world="rema")

Related Guidelines

• Encoding Methodology - Complete encoding process
• Machloket Handling - Testing disputes
• Review Standards - Test requirements in review
• Contributing Encodings - Submission workflow