Skip to content
This repository was archived by the owner on Jun 3, 2025. It is now read-only.

B kosis products #134

Merged
merged 24 commits into from
Dec 9, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
1f5b0ab
Update deepsparse.mdx
Beth-Kosis Dec 6, 2022
3a07722
Update community.mdx
Beth-Kosis Dec 6, 2022
22d38c6
Update enterprise.mdx
Beth-Kosis Dec 6, 2022
5b2f019
Update community.mdx
Beth-Kosis Dec 6, 2022
b322bba
Update community.mdx
Beth-Kosis Dec 6, 2022
1e8162a
Update community.mdx
Beth-Kosis Dec 6, 2022
c9f18bb
Update enterprise.mdx
Beth-Kosis Dec 6, 2022
d888f22
Update sparseml.mdx
Beth-Kosis Dec 6, 2022
67a8f01
Update sparsezoo.mdx
Beth-Kosis Dec 6, 2022
3868881
Update enterprise.mdx
robertgshaw2-redhat Dec 8, 2022
d718234
Update src/content/products/deepsparse/community.mdx
jeanniefinks Dec 8, 2022
02e940f
Update src/content/products/deepsparse/community.mdx
jeanniefinks Dec 8, 2022
8e1f0d0
Update src/content/products/deepsparse/community.mdx
jeanniefinks Dec 8, 2022
245f082
Update src/content/products/deepsparse/enterprise.mdx
jeanniefinks Dec 8, 2022
3d83ecb
Update src/content/products/deepsparse/enterprise.mdx
jeanniefinks Dec 8, 2022
bb6ae51
Update src/content/products/deepsparse/enterprise.mdx
jeanniefinks Dec 8, 2022
aa43b9c
Update src/content/products/deepsparse/enterprise.mdx
jeanniefinks Dec 8, 2022
08514b2
Update src/content/products/deepsparse/enterprise.mdx
jeanniefinks Dec 9, 2022
497bae7
Update src/content/products/deepsparse/enterprise.mdx
jeanniefinks Dec 9, 2022
d08294f
Update src/content/products/sparsezoo.mdx
robertgshaw2-redhat Dec 9, 2022
374a9ad
Update src/content/products/deepsparse/community.mdx
robertgshaw2-redhat Dec 9, 2022
ca7d534
Update src/content/products/deepsparse/community.mdx
robertgshaw2-redhat Dec 9, 2022
ec36db7
Update src/content/products/deepsparse/enterprise.mdx
robertgshaw2-redhat Dec 9, 2022
bc2311c
Update src/content/products/deepsparse/enterprise.mdx
robertgshaw2-redhat Dec 9, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/content/products/deepsparse.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ index: 1000
</div>
</div>

A CPU runtime that takes advantage of sparsity within neural networks to reduce compute. Read [more about sparsification](https://docs.neuralmagic.com/user-guide/sparsification).
DeepSparse is a CPU runtime that takes advantage of sparsity within neural networks to reduce compute. Read [more about sparsification](https://docs.neuralmagic.com/user-guide/sparsification).

Neural Magic's DeepSparse Engine is able to integrate into popular deep learning libraries (e.g., Hugging Face, Ultralytics) allowing you to leverage DeepSparse for loading and deploying sparse models with ONNX.
ONNX gives the flexibility to serve your model in a framework-agnostic environment.
Expand Down
107 changes: 45 additions & 62 deletions src/content/products/deepsparse/community.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,47 +42,34 @@ index: 1000
</div>
</div>

A CPU runtime that takes advantage of sparsity within neural networks to reduce compute. Read more about sparsification Read more about sparsification [here](https://docs.neuralmagic.com/user-guide/sparsification).
<p><br></p>
DeepSparse Community Edition is open-source and free for evaluation, research, and non-production use with our [Engine Community License](https://neuralmagic.com/legal/engine-license-agreement/). (Alternatively, the [Enterprise Edition](https://docs.neuralmagic.com/products/deepsparse-ent) requires a Trial License or can be fully licensed for production, commercial applications.)

Neural Magic's DeepSparse Engine is able to integrate into popular deep learning libraries (e.g., Hugging Face, Ultralytics) allowing you to leverage DeepSparse for loading and deploying sparse models with ONNX.
ONNX gives the flexibility to serve your model in a framework-agnostic environment.
Support includes [PyTorch,](https://pytorch.org/docs/stable/onnx.html) [TensorFlow,](https://github.com/onnx/tensorflow-onnx) [Keras,](https://github.com/onnx/keras-onnx) and [many other frameworks](https://github.com/onnx/onnxmltools).

The DeepSparse Engine is available in two editions:
1. [**The Community Edition**](#installation) is open-source and free for evaluation, research, and non-production use with our [Engine Community License](https://neuralmagic.com/legal/engine-license-agreement/).
2. [**The Enterprise Edition**](https://docs.neuralmagic.com/products/deepsparse-ent) requires a Trial License or [can be fully licensed](https://neuralmagic.com/legal/master-software-license-and-service-agreement/) for production, commercial applications.

## Features

- 🔌 [DeepSparse Server](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/server)
- 📜 [DeepSparse Benchmark](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/benchmark)
- 👩‍💻 [NLP and Computer Vision Tasks Supported](https://github.com/neuralmagic/deepsparse/tree/main/examples)

## 🧰 Hardware Support and System Requirements
## Hardware Support and System Requirements

Review [CPU Hardware Support for Various Architectures](https://docs.neuralmagic.com/deepsparse/source/hardware.html) to understand system requirements.
The DeepSparse Engine works natively on Linux; Mac and Windows require running Linux in a Docker or virtual machine; it will not run natively on those operating systems.
The DeepSparse Engine works natively on Linux. Mac and Windows require running Linux in a Docker or virtual machine; it will not run natively on those operating systems.

The DeepSparse Engine is tested on Python 3.7-3.10, ONNX 1.5.0-1.12.0, ONNX opset version 11+, and manylinux compliant.
The DeepSparse Engine is tested on Python 3.7-3.10, ONNX 1.5.0-1.12.0, and ONNX opset version 11+. It is manylinux compliant.
Using a [virtual environment](https://docs.python.org/3/library/venv.html) is highly recommended.

## Installation

Install the DeepSparse Community Edition as follows:
Install the DeepSparse Community Edition with `pip`:

```bash
pip install deepsparse
```

See the [DeepSparse Community Installation Page](https://docs.neuralmagic.com/get-started/install/deepsparse) for further installation options.
See the [DeepSparse Community Installation page](https://docs.neuralmagic.com/get-started/install/deepsparse) for further installation options.

To trial or inquire about licensing for DeepSparse Enterprise Edition, see the [DeepSparse Enterprise documentation](https://docs.neuralmagic.com/products/deepsparse-enterprise).
## DeepSparse Community Edition Features

## Features
### DeepSparse Server

### 🔌 DeepSparse Server
The DeepSparse Server allows you to serve models and pipelines from the terminal. The server runs on top of the popular FastAPI web framework and Uvicorn web server.

The DeepSparse Server allows you to serve models and pipelines from the terminal. The server runs on top of the popular FastAPI web framework and Uvicorn web server. Install the server using the following command:
Install the server with `pip`:

```bash
pip install deepsparse[server]
Expand All @@ -98,10 +85,10 @@ deepsparse.server \
--model_path "zoo:nlp/question_answering/bert-base/pytorch/huggingface/squad/12layer_pruned80_quant-none-vnni"
```

To look up arguments run: `deepsparse.server --help`.
To look up arguments, run `deepsparse.server --help`.

#### Multiple Models
To serve multiple models in your deployment you can easily build a `config.yaml`. In the example below, we define two BERT models in our configuration for the question answering task:
To serve multiple models in your deployment, you can easily build a `config.yaml`. In the example below, we define two BERT models in our configuration for the question answering task:

```yaml
num_cores: 1
Expand All @@ -117,16 +104,16 @@ endpoints:
batch_size: 1
```

Finally, after your `config.yaml` file is built, run the server with the config file path as an argument:
Finally, after your `config.yaml` file is built, run the server with the configuration file path as an argument:
```bash
deepsparse.server config config.yaml
```

[Getting Started with the DeepSparse Server](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/server) for more info.
See [Getting Started with the DeepSparse Server](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/server) for more information.

### 📜 DeepSparse Benchmark
### DeepSparse Benchmark

The benchmark tool is available on your CLI to run expressive model benchmarks on the DeepSparse Engine with minimal parameters.
The benchmark tool is available on your CLI to run expressive model benchmarks on the DeepSparse Engine with minimal parameters.

Run `deepsparse.benchmark -h` to look up arguments:

Expand All @@ -144,7 +131,9 @@ deepsparse.benchmark [-h] [-b BATCH_SIZE] [-shapes INPUT_SHAPES]
- Asynchronous (Multi-stream) Scenario


### 👩‍💻 NLP Inference Example
### NLP and Computer Vision Tasks Supported

An NLP inference example is:

```python
from deepsparse import Pipeline
Expand All @@ -160,30 +149,37 @@ qa_pipeline = Pipeline.create(
my_name = qa_pipeline(question="What's my name?", context="My name is Snorlax")
```

NLP Tutorials:
- [Getting Started with Hugging Face Transformers 🤗](https://github.com/neuralmagic/deepsparse/tree/main/examples/huggingface-transformers)
Refer also to [Using Pipelines](https://github.com/neuralmagic/deepsparse/blob/main/src/deepsparse/PIPELINES.md).

- For Image Classification tutorials, see [Image Classification Inference Pipelines](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/image_classification).

Tasks Supported:
- For Object Detection tutorials, see [YOLOv5 Inference Pipelines](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/yolo).

- For Segmentation tutorials, see [YOLACT Inference Pipelines](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/yolact).

- For NLP tutorials, see [Getting Started with Hugging Face Transformers](https://github.com/neuralmagic/deepsparse/tree/main/examples/huggingface-transformers).

Supported NLP tasks include:
- [Token Classification: Named Entity Recognition](https://neuralmagic.com/use-cases/sparse-named-entity-recognition/)
- [Text Classification: Multi-Class](https://neuralmagic.com/use-cases/sparse-multi-class-text-classification/)
- [Text Classification: Binary](https://neuralmagic.com/use-cases/sparse-binary-text-classification/)
- [Text Classification: Sentiment Analysis](https://neuralmagic.com/use-cases/sparse-sentiment-analysis/)
- [Question Answering](https://neuralmagic.com/use-cases/sparse-question-answering/)

### 🦉 SparseZoo ONNX vs. Custom ONNX Models
### SparseZoo ONNX vs. Custom ONNX Models

DeepSparse can accept ONNX models from two sources:

- **SparseZoo ONNX**: our open-source collection of sparse models available for download. [SparseZoo](https://github.com/neuralmagic/sparsezoo) hosts inference-optimized models, trained on repeatable sparsification recipes using state-of-the-art techniques from [SparseML](https://github.com/neuralmagic/sparseml).
- **SparseZoo ONNX**: [SparseZoo](https://github.com/neuralmagic/sparsezoo) hosts open-source inference-optimized models, trained on repeatable sparsification recipes using state-of-the-art techniques from [SparseML](https://github.com/neuralmagic/sparseml). The ONNX representation of each model is available for download.

- **Custom ONNX**: your own ONNX model, can be dense or sparse. Plug in your model to compare performance with other solutions.
- **Custom ONNX**: DeepSparse allows you to use your own model in ONNX format. It can be dense or sparse. Plug in your model to compare performance with other solutions.

```bash
> wget https://github.com/onnx/models/raw/main/vision/classification/mobilenet/model/mobilenetv2-7.onnx
Saving to: ‘mobilenetv2-7.onnx’
```

Custom ONNX Benchmark example:
Here is a custom ONNX benchmark example:
```python
from deepsparse import compile_model
from deepsparse.utils import generate_random_inputs
Expand All @@ -202,46 +198,35 @@ The [GitHub repository](https://github.com/neuralmagic/deepsparse) includes pack

### Scheduling Single-Stream, Multi-Stream, and Elastic Inference

The DeepSparse Engine offers up to three types of inferences based on your use case. Read more details here: [Inference Types](https://github.com/neuralmagic/deepsparse/blob/main/docs/source/scheduler.md).

1 ⚡ Single-stream scheduling: the latency/synchronous scenario, requests execute serially. [`default`]

<img src="https://raw.githubusercontent.com/neuralmagic/deepsparse/main/docs/source/single-stream.png" alt="single stream diagram" />
The DeepSparse Engine offers up to three types of inferences based on your use case. Refer also to [Inference Types](https://github.com/neuralmagic/deepsparse/blob/main/docs/source/scheduler.md).

Use Case: It's highly optimized for minimum per-request latency, using all of the system's resources provided to it on every request it gets.
1. Single-stream scheduling (the default) is the latency/synchronous scenario. Requests execute serially.
<br><img src="https://raw.githubusercontent.com/neuralmagic/deepsparse/main/docs/source/single-stream.png" alt="single stream diagram" />
<br>Use Case: It's highly optimized for minimum per-request latency, using all of the system's resources provided to it on every request it gets.

2 ⚡ Multi-stream scheduling: the throughput/asynchronous scenario, requests execute in parallel.
2. Multi-stream scheduling is the throughput/asynchronous scenario. Requests execute in parallel.
<br><img src="https://raw.githubusercontent.com/neuralmagic/deepsparse/main/docs/source/multi-stream.png" alt="multi stream diagram" />
<br>Use Case: The most common use cases for the multi-stream scheduler are those in which parallelism is low with respect to core count, and requests need to be made asynchronously without time to batch them.

<img src="https://raw.githubusercontent.com/neuralmagic/deepsparse/main/docs/source/multi-stream.png" alt="multi stream diagram" />

PRO TIP: The most common use cases for the multi-stream scheduler are where parallelism is low with respect to core count, and where requests need to be made asynchronously without time to batch them.

3 ⚡ Elastic scheduling: requests execute in parallel, but not multiplexed on individual NUMA nodes.

Use Case: A workload that might benefit from the elastic scheduler is one in which multiple requests need to be handled simultaneously, but where performance is hindered when those requests have to share an L3 cache.
3. Elastic scheduling requests execute in parallel, but not multiplexed on individual NUMA nodes.
<br>Use Case: A workload that might benefit from the elastic scheduler is one in which multiple requests need to be handled simultaneously, but where performance is hindered when those requests have to share an L3 cache.

## Resources
#### Libraries
- [DeepSparse](https://docs.neuralmagic.com/deepsparse/)

- [SparseML](https://docs.neuralmagic.com/sparseml/)

- [SparseZoo](https://docs.neuralmagic.com/sparsezoo/)

- [Sparsify](https://docs.neuralmagic.com/sparsify/)


#### Versions
- [DeepSparse](https://pypi.org/project/deepsparse) | stable

- [DeepSparse-Nightly](https://pypi.org/project/deepsparse-nightly/) | nightly (dev)

- [GitHub](https://github.com/neuralmagic/deepsparse/releases) | releases

#### Info

- [Blog](https://www.neuralmagic.com/blog/)

- [Resources](https://www.neuralmagic.com/resources/)


Expand All @@ -252,16 +237,14 @@ Use Case: A workload that might benefit from the elastic scheduler is one in whi

Contribute with code, examples, integrations, and documentation as well as bug reports and feature requests! [Learn how here.](https://github.com/neuralmagic/deepsparse/blob/main/CONTRIBUTING.md)

For user help or questions about DeepSparse, sign up or log in to our **[Deep Sparse Community Slack](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)**. We are growing the community member by member and happy to see you there. Bugs, feature requests, or additional questions can also be posted to our [GitHub Issue Queue.](https://github.com/neuralmagic/deepsparse/issues) You can get the latest news, webinar and event invites, research papers, and other ML Performance tidbits by [subscribing](https://neuralmagic.com/subscribe/) to the Neural Magic community.
For user help or questions about DeepSparse, sign up or log into our [Deep Sparse Community Slack](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ). We are growing the community member by member and happy to see you there. Bugs, feature requests, or additional questions can also be posted to our [GitHub Issue Queue.](https://github.com/neuralmagic/deepsparse/issues) You can get the latest news, webinar and event invites, research papers, and other ML performance tidbits by [subscribing](https://neuralmagic.com/subscribe/) to the Neural Magic community.

For more general questions about Neural Magic, complete this [form.](http://neuralmagic.com/contact/)

### License

DeepSparse Community is licensed under the [Neural Magic DeepSparse Community License.](https://github.com/neuralmagic/deepsparse/blob/main/LICENSE-NEURALMAGIC)
Some source code, example files, and scripts included in the deepsparse GitHub repository or directory are licensed under the [Apache License Version 2.0](https://github.com/neuralmagic/deepsparse/blob/main/LICENSE) as noted.

[DeepSparse Enterprise](https://docs.neuralmagic.com/products/deepsparse-ent) requires a Trial License or [can be fully licensed](https://neuralmagic.com/legal/master-software-license-and-service-agreement/) for production, commercial applications.
Some source code, example files, and scripts included in the DeepSparse GitHub repository or directory are licensed under the [Apache License Version 2.0](https://github.com/neuralmagic/deepsparse/blob/main/LICENSE), as noted.

### Cite

Expand Down
Loading