You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jun 3, 2025. It is now read-only.
Copy file name to clipboardExpand all lines: src/content/products/deepsparse.mdx
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -42,7 +42,7 @@ index: 1000
42
42
</div>
43
43
</div>
44
44
45
-
A CPU runtime that takes advantage of sparsity within neural networks to reduce compute. Read [more about sparsification](https://docs.neuralmagic.com/user-guide/sparsification).
45
+
DeepSparse is a CPU runtime that takes advantage of sparsity within neural networks to reduce compute. Read [more about sparsification](https://docs.neuralmagic.com/user-guide/sparsification).
46
46
47
47
Neural Magic's DeepSparse Engine is able to integrate into popular deep learning libraries (e.g., Hugging Face, Ultralytics) allowing you to leverage DeepSparse for loading and deploying sparse models with ONNX.
48
48
ONNX gives the flexibility to serve your model in a framework-agnostic environment.
Copy file name to clipboardExpand all lines: src/content/products/deepsparse/community.mdx
+45-62Lines changed: 45 additions & 62 deletions
Original file line number
Diff line number
Diff line change
@@ -42,47 +42,34 @@ index: 1000
42
42
</div>
43
43
</div>
44
44
45
-
A CPU runtime that takes advantage of sparsity within neural networks to reduce compute. Read more about sparsification Read more about sparsification [here](https://docs.neuralmagic.com/user-guide/sparsification).
45
+
<p><br></p>
46
+
DeepSparse Community Edition is open-source and free for evaluation, research, and non-production use with our [Engine Community License](https://neuralmagic.com/legal/engine-license-agreement/). (Alternatively, the [Enterprise Edition](https://docs.neuralmagic.com/products/deepsparse-ent) requires a Trial License or can be fully licensed for production, commercial applications.)
46
47
47
-
Neural Magic's DeepSparse Engine is able to integrate into popular deep learning libraries (e.g., Hugging Face, Ultralytics) allowing you to leverage DeepSparse for loading and deploying sparse models with ONNX.
48
-
ONNX gives the flexibility to serve your model in a framework-agnostic environment.
49
-
Support includes [PyTorch,](https://pytorch.org/docs/stable/onnx.html)[TensorFlow,](https://github.com/onnx/tensorflow-onnx)[Keras,](https://github.com/onnx/keras-onnx) and [many other frameworks](https://github.com/onnx/onnxmltools).
50
-
51
-
The DeepSparse Engine is available in two editions:
52
-
1.[**The Community Edition**](#installation) is open-source and free for evaluation, research, and non-production use with our [Engine Community License](https://neuralmagic.com/legal/engine-license-agreement/).
53
-
2.[**The Enterprise Edition**](https://docs.neuralmagic.com/products/deepsparse-ent) requires a Trial License or [can be fully licensed](https://neuralmagic.com/legal/master-software-license-and-service-agreement/) for production, commercial applications.
- 👩💻 [NLP and Computer Vision Tasks Supported](https://github.com/neuralmagic/deepsparse/tree/main/examples)
60
-
61
-
## 🧰 Hardware Support and System Requirements
48
+
## Hardware Support and System Requirements
62
49
63
50
Review [CPU Hardware Support for Various Architectures](https://docs.neuralmagic.com/deepsparse/source/hardware.html) to understand system requirements.
64
-
The DeepSparse Engine works natively on Linux; Mac and Windows require running Linux in a Docker or virtual machine; it will not run natively on those operating systems.
51
+
The DeepSparse Engine works natively on Linux. Mac and Windows require running Linux in a Docker or virtual machine; it will not run natively on those operating systems.
65
52
66
-
The DeepSparse Engine is tested on Python 3.7-3.10, ONNX 1.5.0-1.12.0, ONNX opset version 11+, and manylinux compliant.
53
+
The DeepSparse Engine is tested on Python 3.7-3.10, ONNX 1.5.0-1.12.0, and ONNX opset version 11+. It is manylinux compliant.
67
54
Using a [virtual environment](https://docs.python.org/3/library/venv.html) is highly recommended.
68
55
69
56
## Installation
70
57
71
-
Install the DeepSparse Community Edition as follows:
58
+
Install the DeepSparse Community Edition with `pip`:
72
59
73
60
```bash
74
61
pip install deepsparse
75
62
```
76
63
77
-
See the [DeepSparse Community Installation Page](https://docs.neuralmagic.com/get-started/install/deepsparse) for further installation options.
64
+
See the [DeepSparse Community Installation page](https://docs.neuralmagic.com/get-started/install/deepsparse) for further installation options.
78
65
79
-
To trial or inquire about licensing for DeepSparse Enterprise Edition, see the [DeepSparse Enterprise documentation](https://docs.neuralmagic.com/products/deepsparse-enterprise).
66
+
## DeepSparse Community Edition Features
80
67
81
-
##Features
68
+
### DeepSparse Server
82
69
83
-
### 🔌 DeepSparse Server
70
+
The DeepSparse Server allows you to serve models and pipelines from the terminal. The server runs on top of the popular FastAPI web framework and Uvicorn web server.
84
71
85
-
The DeepSparse Server allows you to serve models and pipelines from the terminal. The server runs on top of the popular FastAPI web framework and Uvicorn web server. Install the server using the following command:
To look up arguments run:`deepsparse.server --help`.
88
+
To look up arguments, run `deepsparse.server --help`.
102
89
103
90
#### Multiple Models
104
-
To serve multiple models in your deployment you can easily build a `config.yaml`. In the example below, we define two BERT models in our configuration for the question answering task:
91
+
To serve multiple models in your deployment, you can easily build a `config.yaml`. In the example below, we define two BERT models in our configuration for the question answering task:
105
92
106
93
```yaml
107
94
num_cores: 1
@@ -117,16 +104,16 @@ endpoints:
117
104
batch_size: 1
118
105
```
119
106
120
-
Finally, after your `config.yaml` file is built, run the server with the config file path as an argument:
107
+
Finally, after your `config.yaml` file is built, run the server with the configuration file path as an argument:
121
108
```bash
122
109
deepsparse.server config config.yaml
123
110
```
124
111
125
-
[Getting Started with the DeepSparse Server](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/server) for more info.
112
+
See [Getting Started with the DeepSparse Server](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/server) for more information.
126
113
127
-
### 📜 DeepSparse Benchmark
114
+
### DeepSparse Benchmark
128
115
129
-
The benchmark tool is available on your CLI to run expressive model benchmarks on the DeepSparse Engine with minimal parameters.
116
+
The benchmark tool is available on your CLI to run expressive model benchmarks on the DeepSparse Engine with minimal parameters.
130
117
131
118
Run `deepsparse.benchmark -h` to look up arguments:
my_name = qa_pipeline(question="What's my name?", context="My name is Snorlax")
161
150
```
162
151
163
-
NLP Tutorials:
164
-
- [Getting Started with Hugging Face Transformers 🤗](https://github.com/neuralmagic/deepsparse/tree/main/examples/huggingface-transformers)
152
+
Refer also to [Using Pipelines](https://github.com/neuralmagic/deepsparse/blob/main/src/deepsparse/PIPELINES.md).
153
+
154
+
- For Image Classification tutorials, see [Image Classification Inference Pipelines](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/image_classification).
165
155
166
-
Tasks Supported:
156
+
- For Object Detection tutorials, see [YOLOv5 Inference Pipelines](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/yolo).
157
+
158
+
- For Segmentation tutorials, see [YOLACT Inference Pipelines](https://github.com/neuralmagic/deepsparse/tree/main/src/deepsparse/yolact).
159
+
160
+
- For NLP tutorials, see [Getting Started with Hugging Face Transformers](https://github.com/neuralmagic/deepsparse/tree/main/examples/huggingface-transformers).
161
+
162
+
Supported NLP tasks include:
167
163
- [Token Classification: Named Entity Recognition](https://neuralmagic.com/use-cases/sparse-named-entity-recognition/)
DeepSparse can accept ONNX models from two sources:
176
172
177
-
- **SparseZoo ONNX**: our open-source collection of sparse models available for download. [SparseZoo](https://github.com/neuralmagic/sparsezoo) hosts inference-optimized models, trained on repeatable sparsification recipes using state-of-the-art techniques from [SparseML](https://github.com/neuralmagic/sparseml).
173
+
- **SparseZoo ONNX**: [SparseZoo](https://github.com/neuralmagic/sparsezoo) hosts open-source inference-optimized models, trained on repeatable sparsification recipes using state-of-the-art techniques from [SparseML](https://github.com/neuralmagic/sparseml). The ONNX representation of each model is available for download.
178
174
179
-
- **Custom ONNX**: your own ONNX model, can be dense or sparse. Plug in your model to compare performance with other solutions.
175
+
- **Custom ONNX**: DeepSparse allows you to use your own model in ONNX format. It can be dense or sparse. Plug in your model to compare performance with other solutions.
from deepsparse.utils import generate_random_inputs
@@ -202,46 +198,35 @@ The [GitHub repository](https://github.com/neuralmagic/deepsparse) includes pack
202
198
203
199
### Scheduling Single-Stream, Multi-Stream, and Elastic Inference
204
200
205
-
The DeepSparse Engine offers up to three types of inferences based on your use case. Read more details here: [Inference Types](https://github.com/neuralmagic/deepsparse/blob/main/docs/source/scheduler.md).
The DeepSparse Engine offers up to three types of inferences based on your use case. Refer also to [Inference Types](https://github.com/neuralmagic/deepsparse/blob/main/docs/source/scheduler.md).
210
202
211
-
Use Case: It's highly optimized for minimum per-request latency, using all of the system's resources provided to it on every request it gets.
203
+
1. Single-stream scheduling (the default) is the latency/synchronous scenario. Requests execute serially.
<br>Use Case: The most common use cases for the multi-stream scheduler are those in which parallelism is low with respect to core count, and requests need to be made asynchronously without time to batch them.
PRO TIP: The most common use cases for the multi-stream scheduler are where parallelism is low with respect to core count, and where requests need to be made asynchronously without time to batch them.
218
-
219
-
3 ⚡ Elastic scheduling: requests execute in parallel, but not multiplexed on individual NUMA nodes.
220
-
221
-
Use Case: A workload that might benefit from the elastic scheduler is one in which multiple requests need to be handled simultaneously, but where performance is hindered when those requests have to share an L3 cache.
211
+
3. Elastic scheduling requests execute in parallel, but not multiplexed on individual NUMA nodes.
212
+
<br>Use Case: A workload that might benefit from the elastic scheduler is one in which multiple requests need to be handled simultaneously, but where performance is hindered when those requests have to share an L3 cache.
@@ -252,16 +237,14 @@ Use Case: A workload that might benefit from the elastic scheduler is one in whi
252
237
253
238
Contribute with code, examples, integrations, and documentation as well as bug reports and feature requests! [Learn how here.](https://github.com/neuralmagic/deepsparse/blob/main/CONTRIBUTING.md)
254
239
255
-
For user help or questions about DeepSparse, sign up or log in to our **[Deep Sparse Community Slack](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)**. We are growing the community member by member and happy to see you there. Bugs, feature requests, or additional questions can also be posted to our [GitHub Issue Queue.](https://github.com/neuralmagic/deepsparse/issues) You can get the latest news, webinar and event invites, research papers, and other ML Performance tidbits by [subscribing](https://neuralmagic.com/subscribe/) to the Neural Magic community.
240
+
For user help or questions about DeepSparse, sign up or log into our [Deep Sparse Community Slack](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ). We are growing the community member by member and happy to see you there. Bugs, feature requests, or additional questions can also be posted to our [GitHub Issue Queue.](https://github.com/neuralmagic/deepsparse/issues) You can get the latest news, webinar and event invites, research papers, and other ML performance tidbits by [subscribing](https://neuralmagic.com/subscribe/) to the Neural Magic community.
256
241
257
242
For more general questions about Neural Magic, complete this [form.](http://neuralmagic.com/contact/)
258
243
259
244
### License
260
245
261
246
DeepSparse Community is licensed under the [Neural Magic DeepSparse Community License.](https://github.com/neuralmagic/deepsparse/blob/main/LICENSE-NEURALMAGIC)
262
-
Some source code, example files, and scripts included in the deepsparse GitHub repository or directory are licensed under the [Apache License Version 2.0](https://github.com/neuralmagic/deepsparse/blob/main/LICENSE) as noted.
263
-
264
-
[DeepSparse Enterprise](https://docs.neuralmagic.com/products/deepsparse-ent) requires a Trial License or [can be fully licensed](https://neuralmagic.com/legal/master-software-license-and-service-agreement/) for production, commercial applications.
247
+
Some source code, example files, and scripts included in the DeepSparse GitHub repository or directory are licensed under the [Apache License Version 2.0](https://github.com/neuralmagic/deepsparse/blob/main/LICENSE), as noted.
0 commit comments