Polars पाइपलाइन का परीक्षण
Polars के साथ Data Pipelines का स्केलिंग और ऑप्टिमाइज़ेशन
Liam Brannigan
Data Scientist & Polars Contributor
पाइपलाइन का परीक्षण क्यों?
पाइपलाइन का परीक्षण क्यों?
विभाग अनुसार पूर्ण अनुरोध
def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
return
विभाग अनुसार पूर्ण अनुरोध
def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
return (
requests
)
विभाग अनुसार पूर्ण अनुरोध
def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
return (
requests
.group_by("DEPARTMENT")
.len()
)
विभाग अनुसार पूर्ण अनुरोध
def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
return (
requests
.group_by("DEPARTMENT")
.len()
.with_columns(pl.col("len").cast(pl.Int32))
.sort("DEPARTMENT")
)
टेस्ट इनपुट
sample = pl.DataFrame({
"SR_NUMBER": ["SR1", "SR2", "SR3", "SR4"],
"DEPARTMENT": ["Sanitation", "Water", "Sanitation", "Aviation"],
"STATUS": ["Completed", "Open", "Completed", "Completed"],
})
shape: (4, 3)
| SR_NUMBER | DEPARTMENT | STATUS |
| --- | --- | --- |
| str | str | str |
|-----------|------------|-----------|
| SR1 | Sanitation | Completed |
| SR2 | Water | Open |
| SR3 | Sanitation | Completed |
| SR4 | Aviation | Completed |
अपेक्षित आउटपुट
expected = pl.DataFrame(
{
"DEPARTMENT": ["Aviation", "Sanitation"],
"len": [1, 2],
}
)
shape: (2, 2)
| DEPARTMENT | len |
| --- | --- |
| str | i64 |
|------------|-----|
| Aviation | 1 |
| Sanitation | 2 |
वास्तविक परिणाम निकालना
actual = completed_by_department(
वास्तविक परिणाम निकालना
actual = completed_by_department(
sample.lazy()
).collect()
वास्तविक परिणाम निकालना
actual = completed_by_department(
sample.lazy()
).collect()
shape: (3, 2)
| DEPARTMENT | len |
| --- | --- |
| str | i32 |
|------------|-----|
| Aviation | 1 |
| Sanitation | 2 |
| Water | 1 |
equals से तुलना
actual.equals(expected)
False
Polars testing इम्पोर्ट करना
from polars.testing import (
assert_frame_equal,
assert_schema_equal,
)
स्कीमा का परीक्षण
assert_schema_equal(
)
स्कीमा का परीक्षण
assert_schema_equal(
actual.schema,
expected.schema,
)
AssertionError: Schemas are different (dtypes do not match)
[left]: [String, Int32]
[right]: [String, Int64]
अपेक्षित स्कीमा ठीक करना
expected = pl.DataFrame(
{
"DEPARTMENT": ["Aviation", "Sanitation"], "len": [1, 2],
},
schema={"DEPARTMENT": pl.String, "len": pl.Int32},
)
shape: (2, 2)
| DEPARTMENT | len |
| --- | --- |
| str | i32 |
|------------|-----|
| Aviation | 1 |
| Sanitation | 2 |
स्कीमा का परीक्षण
assert_schema_equal(
actual.schema,
expected.schema,
)
print("Schema test passed!")
Schema test passed!
फ्रेम का परीक्षण
assert_frame_equal(
)
फ्रेम का परीक्षण
assert_frame_equal(
actual,
expected,
)
AssertionError: DataFrames are different height (row count) mismatch
[left]: 3
[right]: 2
वास्तविक बनाम अपेक्षित तुलना
shape: (3, 2)
| DEPARTMENT | len |
| --- | --- |
| str | i32 |
|------------|-----|
| Aviation | 1 |
| Sanitation | 2 |
| Water | 1 |
shape: (2, 2)
| DEPARTMENT | len |
| --- | --- |
| str | i32 |
|------------|-----|
| Aviation | 1 |
| Sanitation | 2 |
क्वेरी ठीक करना
def completed_by_department(requests: pl.LazyFrame) -> pl.LazyFrame:
return (
requests
.filter(pl.col("STATUS") == "Completed")
.group_by("DEPARTMENT")
.len()
.with_columns(pl.col("len").cast(pl.Int32))
.sort("DEPARTMENT")
)
actual = completed_by_department(
sample.lazy()
).collect()
अंतिम असेर्शन
assert_schema_equal(actual.schema, expected.schema)
assert_frame_equal(actual, expected)
print("All tests passed!")
All tests passed!
अभ्यास करते हैं!
Polars के साथ Data Pipelines का स्केलिंग और ऑप्टिमाइज़ेशन
