Introduction to Data Quality with Great Expectations
Davina Moossazadeh
Data Scientist
expectation = gx.expectations.ExpectTableColumnCountToEqual(
value=10
)
suite = gx.ExpectationSuite(
name="my_suite"
)
# Add Expectation to Suite
suite.add_expectation(
expectation=expectation
)
# Create another Expectation Suite
another_suite = gx.ExpectationSuite(name="my_other_suite")
# Add the same Expectation to the new Suite
another_suite.add_expectation(expectation=expectation)
Expectations cannot belong to multiple Suites at once:
RuntimeError: Cannot add Expectation because it already belongs to an
ExpectationSuite. If you want to update an existing Expectation, please call
Expectation.save(). If you are copying this Expectation to a new ExpectationSuite,
please copy it first (the core expectations and some others support
copy(expectation)) and set `Expectation.id = None`.
If you are copying this Expectation to a new ExpectationSuite, please copy it first
(the core expectations and some others support copy(expectation)) and set
`Expectation.id = None`.
Copy the Expectation, set its .id
to None
, and add it to the new Suite without errors:
expectation_copy = expectation.copy()
expectation_copy.id = None
another_suite.add_expectation(
expectation=expectation_copy
)
print(
expectation_copy in another_suite.expectations
)
True
Add
.add_expectation()
suite.add_expectation(
expectation=expectation
)
Delete
.delete_expectation()
suite.delete_expectation(
expectation=expectation
)
Update the .value
attribute and save changes:
expectation = gx.expectations.ExpectTableColumnCountToEqual( value=10 )
expectation.value = 11
expectation.save()
Ensure the Expectation belongs to a Suite, otherwise:
RuntimeError: Expectation must be added to ExpectationSuite before it can be saved.
suite = gx.ExpectationSuite(name="my_suite")
validation_definition = gx.ValidationDefinition(
data=batch_definition, suite=suite, name="my_validation_definition"
)
# Define Expectation col_name_expectation = gx.expectations.ExpectColumnToExist(column="GHI")
# Add Expectation to Suite suite.add_expectation(expectation=col_name_expectation)
# Run Validation Definition associated with the Suite validation_results = validation_definition.run()
Save changes to the Suite before running the Validation Definition to avoid errors:
validation_results = validation_definition.run()
ResourceFreshnessAggregateError: ExpectationSuite 'my_suite' has changed since it
has last been saved. Please update with `<SUITE_OBJECT>.save()`, then try your
action again.
Use the .save()
method to save the Suite and run the Validation Definition error-free:
suite.save()
validation_results = validation_definition.run()
print(validation_results.success)
False
Copy Expectation:
expectation_copy = expectation.copy()
expectation_copy.id = None
Check if Expectation is in Suite:
expectation in suite.expectations
Delete Expectation:
suite.delete_expectation(expectation)
Update Expectation value:
expectation.value = new_value
Save changes to Expectation:
expectation.save()
Save changes to Expectation Suite:
suite.save()
Introduction to Data Quality with Great Expectations