Deepchecks - Continuous Validation for AI & ML: Testing, CI & Monitoring
Deepchecks is a holistic open-source solution for all of your AI & ML validation needs,
enabling you to thoroughly test your data and models from research to production.
## 🧩 Components
Deepchecks includes:
- **Deepchecks Testing**
([Quickstart](#---------deepchecks-testing-quickstart------),
[docs](https://docs.deepchecks.com/stable/?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=components)):
- Running built-in & your own custom Checks and Suites for Tabular, NLP & CV validation (open source).
- **CI & Testing Management**
([Quickstart](#---------deepchecks-ci--testing-management-quickstart------),
[docs](https://docs.deepchecks.com/stable/general/usage/ci_cd.html?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=components)):
- Collaborating over test results and iterating efficiently until
model is production-ready and can be deployed (open source & managed offering).
- **Deepchecks Monitoring**
([Quickstart](#---------deepchecks-monitoring-quickstart------),
[docs](https://docs.deepchecks.com/monitoring/stable/?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=components)):
- Tracking and validating your deployed models behavior when in production (open source & managed offering).
This repo is our main repo as all components use the deepchecks checks in their core. See the [Getting Started](#--getting-started) section for more information about installation and quickstarts for each of the components.
If you want to see deepchecks monitoring's code, you can check out the
[deepchecks/monitoring](https://github.com/deepchecks/monitoring) repo.
## ⏩ Getting Started
💻 Installation
#### Deepchecks Testing (and CI) Installation
```bash
pip install deepchecks -U --user
```
For installing the nlp / vision submodules or with conda:
- For NLP: Replace ``deepchecks`` with ``"deepchecks[nlp]"``,
and optionally install also``deepchecks[nlp-properties]``
- For Computer Vision: Replace ``deepchecks`` with ``"deepchecks[vision]"``.
- For installing with conda, similarly use: ``conda install -c conda-forge deepchecks``.
Check out the full installation instructions for deepchecks testing [here](https://docs.deepchecks.com/stable/getting-started/installation.html).
#### Deepchecks Monitoring Installation
To use deepchecks for production monitoring, you can either use our SaaS service, or deploy a local instance in one line on Linux/MacOS (Windows is WIP!) with Docker.
Create a new directory for the installation files, open a terminal within that directory and run the following:
```
pip install deepchecks-installer
deepchecks-installer install-monitoring
```
This will automatically download the necessary dependencies, run the installation process
and then start the application locally.
The installation will take a few minutes. Then you can open the deployment url (default is http://localhost),
and start the system onboarding. Check out the full monitoring [open source installation & quickstart](https://docs.deepchecks.com/monitoring/stable/getting-started/deploy_self_host_open_source.html).
Note that the open source product is built such that each deployment supports monitoring of
a single model.
### 🏃♀️ Quickstarts
Deepchecks Testing Quickstart
Jump right into the respective quickstart docs:
- [Tabular Quickstarts](https://docs.deepchecks.com/stable/tabular/auto_tutorials/quickstarts/index.html)
- [NLP Quickstarts](https://docs.deepchecks.com/stable/nlp/auto_tutorials/quickstarts/index.html)
- [Vision Quickstarts](https://docs.deepchecks.com/stable/vision/auto_tutorials/quickstarts/index.html)
to have it up and running on your data.
Inside the quickstarts, you'll see how to create the relevant deepchecks object for holding your data and metadata
([Dataset](https://docs.deepchecks.com/stable/tabular/usage_guides/dataset_object.html?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=running_a_suite), [TextData](https://docs.deepchecks.com/stable/nlp/usage_guides/text_data_object.html?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=running_a_suite) or [VisionData](https://docs.deepchecks.com/stable/vision/usage_guides/visiondata_object.html), corresponding to the data type), and run a Suite or Check.
The code snippet for running it will look something like the following, depending on the chosen Suite or Check.
```python
from deepchecks.tabular.suites import model_evaluation
suite = model_evaluation()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_result.save_as_html() # replace this with suite_result.show() or suite_result.show_in_window() to see results inline or in window
# or suite_result.results[0].value with the relevant check index to process the check result's values in python
```
The output will be a report that enables you to inspect the status and results of the chosen checks:
Deepchecks Monitoring Quickstart
Jump right into the
[open source monitoring quickstart docs](https://docs.deepchecks.com/monitoring/stable/getting-started/deploy_self_host_open_source.html)
to have it up and running on your data.
You'll then be able to see the checks results over time, set alerts, and interact
with the dynamic deepchecks UI that looks like this:
Deepchecks CI & Testing Management Quickstart
Deepchecks managed CI & Testing management is currently in closed preview.
[Book a demo](https://deepchecks.com/book-demo/) for more information about the offering.
For building and maintaining your own CI process while utilizing Deepchecks Testing for it,
check out our [docs for Using Deepchecks in CI/CD](https://docs.deepchecks.com/stable/general/usage/ci_cd.html).
## 🧮 How does it work?
At its core, deepchecks includes a wide variety of built-in Checks,
for testing all types of data and model related issues.
These checks are implemented for various models and data types (Tabular, NLP, Vision),
and can easily be customized and expanded.
The check results can be used to automatically make informed decisions
about your model's production-readiness, and for monitoring it over time in production.
The check results can be examined with visual reports (by saving them to an HTML file, or seeing them in Jupyter),
processed with code (using their pythonic / json output), and inspected and collaborated on with Deepchecks' dynamic UI
(for examining test results and for production monitoring).
✅ Deepchecks' Core: The Checks
- All of the Checks and the framework for customizing them are implemented inside
the Deepchecks Testing Python package (this repo).
- Each check tests for a specific potential problem. Deepchecks has many pre-implemented checks for
finding issues with the model's performance (e.g. identifying weak segments),
data distribution (e.g. detect drifts or leakages) and data integrity (e.g. find conflicting labels).
- Customizable: each check has many configurable parameters, and custom checks can easily be implemented.
- Can be run manually (during research) or triggered automatically (in CI processes or production monitoring)
- Check results can be consumed by:
- Visual output report - [Saving to HTML](https://docs.deepchecks.com/stable/general/usage/export_save_results.html)(``result.save_to_html('output_report_name.html')``) or [viewing them](https://docs.deepchecks.com/stable/general/usage/showing_results.html) in Jupyter (``result.show()``).
- Processing with code - with python using the check result's ``value`` attribute, or saving a
[JSON output](https://docs.deepchecks.com/stable/general/usage/export_save_results.html)
- Deepchecks' UI - for dynamic inspection and collaboration (of test results and production monitoring)
- Optional conditions can be added and customized, to automatically validate check results,
with a a pass ✓, fail ✖ or warning ! status
- An ordered list of checks (with optional conditions) can be run together in a "Suite"
(and the output is a concluding report of all checks that ran)
## 📜 Open Source vs Paid
Deepchecks' projects (``deepchecks/deepchecks`` & ``deepchecks/monitoring``) are open source and are released under [AGPL 3.0](./LICENSE).
The only exception are the Deepchecks Monitoring components (in the ``deepchecks/monitoring`` repo), that are under the
([backend/deepchecks_monitoring/ee](https://github.com/deepchecks/monitoring/tree/main/backend/deepchecks_monitoring/ee))
directory, that are subject to a commercial license (see the license [here](https://deepchecks.com/terms-and-conditions)).
That directory isn't used by default, and is packaged as part of the deepchecks monitoring repository simply to
support upgrading to the commercial edition without downtime.
Enabling premium features (contained in the `backend/deepchecks_monitoring/ee` directory) with a self-hosted instance requires a Deepchecks license.
To learn more, [book a demo](https://deepchecks.com/book-demo/) or see our [pricing page](https://deepchecks.com/pricing).
Looking for a 💯% open-source solution for deepcheck monitoring?
Check out the [Monitoring OSS](https://github.com/deepchecks/monitoring-oss) repository, which is purged of all proprietary code and features.
## 👭 Community, Contributing, Docs & Support
Deepchecks is an open source solution.
We are committed to a transparent development process and highly appreciate any contributions.
Whether you are helping us fix bugs, propose new features, improve our documentation or spread the word,
we would love to have you as part of our community.
- Give us a ⭐️ github star ⭐️ on the top of this page to support what we're doing,
it means a lot for open source projects!
- Read our
[docs](https://docs.deepchecks.com/?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=docs)
for more info about how to use and customize deepchecks, and for step-by-step tutorials.
- Post a [Github
Issue](https://github.com/deepchecks/deepchecks/issues) to submit a bug report, feature request, or suggest an improvement.
- To contribute to the package, check out our [first good issues](https://github.com/deepchecks/deepchecks/contribute)
and [contribution guidelines](CONTRIBUTING.rst), and open a PR.
Join our Slack to give us feedback, connect with the maintainers and fellow users, ask questions,
get help for package usage or contributions, or engage in discussions about ML testing!
## ✨ Contributors
Thanks goes to these wonderful people ([emoji
key](https://allcontributors.org/docs/en/emoji-key)):
## ✨ Contributors
Thanks goes to these wonderful people ([emoji
key](https://allcontributors.org/docs/en/emoji-key)):
