Menguji Pipeline Polars

Menskalakan dan Mengoptimalkan Pipeline Data dengan Polars

Liam Brannigan

Data Scientist & Polars Contributor

Mengapa menguji pipeline?

Diagram yang menunjukkan dua DataFrame berbeda pada satu sel.

Mengapa menguji pipeline?

Diagram yang menunjukkan transformasi Polars diuji terhadap DataFrame yang diharapkan.

Permintaan selesai per departemen

def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
    return

Permintaan selesai per departemen

def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
    return (
        requests




    )

Permintaan selesai per departemen

def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
    return (
        requests
        .group_by("DEPARTMENT")
        .len()


    )

Permintaan selesai per departemen

def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
    return (
        requests
        .group_by("DEPARTMENT")
        .len()
        .with_columns(pl.col("len").cast(pl.Int32))
        .sort("DEPARTMENT")
    )

Input uji

sample = pl.DataFrame({
    "SR_NUMBER": ["SR1", "SR2", "SR3", "SR4"],
    "DEPARTMENT": ["Sanitation", "Water", "Sanitation", "Aviation"],
    "STATUS": ["Completed", "Open", "Completed", "Completed"],
})

shape: (4, 3)
| SR_NUMBER | DEPARTMENT | STATUS    |
| ---       | ---        | ---       |
| str       | str        | str       |
|-----------|------------|-----------|
| SR1       | Sanitation | Completed |
| SR2       | Water      | Open      |
| SR3       | Sanitation | Completed |
| SR4       | Aviation   | Completed |

Keluaran yang diharapkan

expected = pl.DataFrame(
    {
        "DEPARTMENT": ["Aviation", "Sanitation"],
        "len": [1, 2],
    }
)

shape: (2, 2)
| DEPARTMENT | len |
| ---        | --- |
| str        | i64 |
|------------|-----|
| Aviation   | 1   |
| Sanitation | 2   |

Mengambil hasil aktual

actual = completed_by_department(

Mengambil hasil aktual

actual = completed_by_department(
    sample.lazy()
).collect()

Mengambil hasil aktual

actual = completed_by_department(
    sample.lazy()
).collect()

shape: (3, 2)
| DEPARTMENT | len |
| ---        | --- |
| str        | i32 |
|------------|-----|
| Aviation   | 1   |
| Sanitation | 2   |
| Water      | 1   |

Membandingkan dengan equals

actual.equals(expected)

False

Mengimpor pengujian Polars

from polars.testing import (
    assert_frame_equal,
    assert_schema_equal,
)

Menguji skema

assert_schema_equal(


)

Menguji skema

assert_schema_equal(
    actual.schema,
    expected.schema,
)

AssertionError: Schemas are different (dtypes do not match)
[left]: [String, Int32]
[right]: [String, Int64]

Memperbaiki skema yang diharapkan

expected = pl.DataFrame(
    {
        "DEPARTMENT": ["Aviation", "Sanitation"], "len": [1, 2],
    },
    schema={"DEPARTMENT": pl.String, "len": pl.Int32},
)

shape: (2, 2)
| DEPARTMENT | len |
| ---        | --- |
| str        | i32 |
|------------|-----|
| Aviation   | 1   |
| Sanitation | 2   |

Menguji skema

assert_schema_equal(
    actual.schema,
    expected.schema,
)
print("Schema test passed!")

Schema test passed!

Menguji frame

assert_frame_equal(


)

Menguji frame

assert_frame_equal(
    actual,
    expected,
)

AssertionError: DataFrames are different height (row count) mismatch
[left]: 3
[right]: 2

Membandingkan aktual dan ekspektasi

shape: (3, 2)
| DEPARTMENT | len |
| ---        | --- |
| str        | i32 |
|------------|-----|
| Aviation   | 1   |
| Sanitation | 2   |
| Water      | 1   |

shape: (2, 2)
| DEPARTMENT | len |
| ---        | --- |
| str        | i32 |
|------------|-----|
| Aviation   | 1   |
| Sanitation | 2   |

Memperbaiki kueri

def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
    return (
        requests
        .filter(pl.col("STATUS") == "Completed")
        .group_by("DEPARTMENT")
        .len()
        .with_columns(pl.col("len").cast(pl.Int32))
        .sort("DEPARTMENT")
    )

actual = completed_by_department(
    sample.lazy()
).collect()

Asersi akhir

assert_schema_equal(actual.schema, expected.schema)
assert_frame_equal(actual, expected)
print("All tests passed!")

All tests passed!

Ayo berlatih!

Menskalakan dan Mengoptimalkan Pipeline Data dengan Polars