Skip to content

Commit d5ff028

Browse files
morteza89morteza89
authored
Add OpenVINO auto-device notebook (#547)
* Add OpenVINO auto-device notebook * Fix E501 formatting failure in notebook_utils.py Co-authored-by: morteza89 <51894034+morteza89@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Avoid shell invocation in notebook pip helper * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Fix python code format job Co-authored-by: morteza89 <51894034+morteza89@users.noreply.github.com> ---------
1 parent 536d134 commit d5ff028

9 files changed

Lines changed: 4385 additions & 1 deletion

File tree

‎AI-Documentation/aidocapp/model.py‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,16 @@ def __init__(
2020

2121
self.template = [{
2222
"role": "user",
23-
"content": "Add a detailed docstring in the style of PEP 257 to the following {} method {}. The docstring should include: - A concise summary of the method's purpose.- A detailed description of each argument (name and type). - A description of the return value (if any). Add inline comments within the method body to explain complex logic or non-obvious steps. Return the method implementation with the docstring and inline comments as a single markdown code block. Do not modify the code. Do not add any chat-like comments."
23+
"content": (
24+
"Add a detailed docstring in the style of PEP 257 to the following "
25+
"{} method {}. The docstring should include: - A concise summary of "
26+
"the method's purpose.- A detailed description of each argument "
27+
"(name and type). - A description of the return value (if any). "
28+
" Add inline comments within the method body to explain complex "
29+
"logic or non-obvious steps. Return the method implementation with "
30+
"the docstring and inline comments as a single markdown code block. "
31+
"Do not modify the code. Do not add any chat-like comments."
32+
),
2433
}]
2534

2635
def generate_comments(self, code, language):

‎openvino_notebooks/auto-device/.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
.venv/
2+
.uv-cache/
3+
__pycache__/
4+
.ipynb_checkpoints/
5+
model/

‎openvino_notebooks/auto-device/.python-version‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
3.12

‎openvino_notebooks/auto-device/README.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# What this notebook does
2+
3+
This notebook shows how to do inference with Automatic Device Selection (AUTO) in OpenVINO and gives a high-level overview of how AUTO chooses the most suitable execution device based on model and hardware availability.
4+
5+
It demonstrates how to compile a model with AUTO, compare first inference latency (model compilation time + first inference time) between GPU and AUTO, and show the difference between THROUGHPUT and LATENCY performance hints.
6+
7+
This notebook provides a practical, engineering-focused introduction to deploying one application across heterogeneous systems (CPU/GPU/NPU) with minimal device-specific branching. It includes explicit device discovery, deterministic fallback behavior, idempotent model preparation/loading, and repeatable runtime measurements for first-inference and steady-state execution.
8+
9+
# Hardware & device support
10+
11+
This notebook supports the following devices:
12+
13+
- CPU — **supported (fallback)**
14+
- GPU — **supported if available**
15+
- NPU — **supported if available**
16+
- AUTO — **primary execution mode**
17+
18+
# Setup
19+
20+
Make sure that **uv** is installed.
21+
22+
## Windows
23+
irm https://astral.sh/uv/install.ps1 | iex
24+
25+
## macOS / Linux
26+
curl -LsSf https://astral.sh/uv/install.sh | sh
27+
28+
Verify installation:
29+
uv --version
30+
31+
From the auto-device notebook folder:
32+
33+
uv sync
34+
35+
uv run jupyter lab auto-device.ipynb
36+
37+
The sample image is included with the notebook. An internet connection is required only on the first run to download the pretrained ResNet-50 weights and create model/resnet50.xml.
38+
39+
# Expected output
40+
41+
When the notebook runs successfully, you should see:
42+
43+
- **Device discovery and selection output**, e.g.
44+
`Available devices: ['CPU', 'GPU.0']`
45+
`Selected device: GPU`
46+
47+
- **Idempotent model preparation logs**, e.g.
48+
`IR model saved to model/resnet50.xml` on first run, then
49+
`Read IR model from model/resnet50.xml` on subsequent runs.
50+
51+
- **Compilation and first-inference timing output**, e.g.
52+
`Time to load model on GPU device and get first inference: 0.15 seconds.`
53+
54+
- **Performance-hint measurements**, with throughput/latency metrics printed over multiple intervals.
55+
56+
For systems without accelerator devices, CPU fallback output is expected and valid.
57+
58+
# Tested-on
59+
60+
| OS | Python | OpenVINO | Device(s) | Status |
61+
|----|--------|----------|-----------|--------|
62+
| Windows 11 | 3.12 | 2026.2 | CPU, GPU | Pass |
63+
64+
# Troubleshooting
65+
66+
### AUTO always selects CPU
67+
**Cause:** GPU/NPU plugin not available, unsupported hardware, or driver/runtime mismatch.
68+
**Fix:** Verify `ov.Core().available_devices`, then update Intel GPU/NPU drivers and confirm execution inside the correct uv environment.
69+
70+
### First inference is much slower than later runs
71+
**Cause:** Model compilation and backend warm-up overhead during first execution.
72+
**Fix:** Compare first-run timing to repeated runs; use multiple iterations for steady-state performance analysis.
73+
74+
### Notebook import errors (openvino/torchvision/notebook utils)
75+
**Cause:** Environment not synced or wrong interpreter selected.
76+
**Fix:** Run `uv sync` in this directory and launch with `uv run jupyter lab auto-device.ipynb`.
77+
78+
### AUTO behavior differs across machines
79+
**Cause:** Different available hardware backends or plugin versions.
80+
**Fix:** Log `core.available_devices` at startup and keep OpenVINO/runtime stack consistent across systems.
81+
82+
# References
83+
84+
- Upstream OpenVINO auto-device notebook:
85+
https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/auto-device/auto-device.ipynb
86+
87+
- OpenVINO AUTO device documentation:
88+
https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/auto-device-selection.html
89+
90+
- OpenVINO performance hints:
91+
https://docs.openvino.ai/2024/openvino-workflow/running-inference/performance-hints.html

0 commit comments

Comments
 (0)