Disable subprocess spawning during hiprtc kernel compilation

[pnikolic-amd]

Motivation

This option is introduced to prevent migraphx to create processes and write to files while comping kernels with hiprtc. Some use cases prevent use of processes so that why this option is introduced. When set kernel compilation will be done sequentially which will result in slower compilation time.

Technical Details

This option can be set using backend_options field in compile_options struct. Example:

migraphx::compile_options options;
options.backend_options["hiprtc_disable_processes"] = migraphx::value(true);
program.compile(migraphx::make_target("gpu"), options);

[pfultz2]

Can this be a cmake option instead?

[gh-app-migraphx-bot-pr-write]

Test	Batch	New Rate (a025c5)	Old Rate (3a503c)*	Diff	Status
torchvision-resnet50	64	3,303.31	3,264.92	1.18%	✅
torchvision-resnet50_fp16	64	7,882.53	7,548.67	4.42%	✅
torchvision-densenet121	32	2,491.44	2,483.99	0.30%	✅
torchvision-densenet121_fp16	32	5,009.13	5,004.24	0.10%	✅
torchvision-inceptionv3	32	2,059.44	2,058.51	0.05%	✅
torchvision-inceptionv3_fp16	32	4,440.94	4,416.99	0.54%	✅
cadene-inceptionv4	16	817.20	820.61	-0.41%	✅
cadene-resnext64x4	16	785.82	782.78	0.39%	✅
slim-mobilenet	64	8,376.00	8,386.36	-0.12%	✅
slim-nasnetalarge	64	229.33	228.86	0.21%	✅
slim-resnet50v2	64	3,228.48	3,180.91	1.50%	✅
bert-mrpc-onnx	8	1,169.35	1,168.84	0.04%	✅
bert-mrpc-tf	1	504.32	498.63	1.14%	✅
pytorch-examples-wlang-gru	1	455.80	473.35	-3.71%	✅
pytorch-examples-wlang-lstm	1	663.52	384.83	72.42%	🔆
torchvision-resnet50_1	1	1,047.98	1,046.63	0.13%	✅
cadene-dpn92_1	1	456.42	437.32	4.37%	✅
cadene-resnext101_1	1	363.33	365.89	-0.70%	✅
onnx-taau-downsample	1	841.77	844.09	-0.28%	✅
dlrm-criteoterabyte	1	32.24	32.42	-0.57%	✅
dlrm-criteoterabyte_fp16	1	51.47	51.80	-0.64%	✅
agentmodel	1	14,718.83	9,209.12	59.83%	🔆
unet_fp16	2	58.37	58.80	-0.73%	✅
resnet50v1_fp16	1	1,445.93	1,366.11	5.84%	🔆
resnet50v1_int8	1	1,764.38	1,883.96	-6.35%	🔴
bert_base_cased_fp16	64	1,098.63	1,098.16	0.04%	✅
bert_large_uncased_fp16	32	347.43	345.59	0.53%	✅
bert_large_fp16	1	206.43	206.59	-0.08%	✅
distilgpt2_fp16	16	2,097.37	2,092.89	0.21%	✅
yolov5s	1	563.28	558.33	0.89%	✅
tinyllama	1	45.82	45.83	-0.03%	✅
vicuna-fastchat	1	44.26	44.20	0.14%	✅
whisper-tiny-encoder	1	413.14	411.87	0.31%	✅
whisper-tiny-decoder	1	409.57	408.48	0.27%	✅
llama2_7b	1	20.86	20.84	0.12%	✅
qwen1.5-7b	1	23.65	23.58	0.32%	✅
phi3-3.8b	1	28.35	26.72	6.11%	🔆
llama3-8b	1	22.74	21.80	4.27%	✅
whisper-large-encoder	1	10.17	10.18	-0.10%	✅
whisper-large-decoder	1	106.46	105.30	1.10%	✅
mistral-7b	1	23.79	23.78	0.06%	✅
FLUX.1-schnell	1	787.76	755.22	4.31%	✅

Regressions detected 🔴

* No develop baseline was found for this PR's branch point; compared against the latest available develop run instead.

[gh-app-migraphx-bot-pr-write]

Test	Status	Result
bert-mrpc-onnx	✅	PASSED: MIGraphX meets tolerance
bert-mrpc-tf	❌	ERROR - check error output traceback Traceback (most recent call last): File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 377, in main() File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 313, in main import tensorflow as tf File "/usr/local/lib/python3.12/dist-packages/tensorflow/init.py", line 40, in from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow # pylint: disable=unused-import ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 37, in self_check.preload_check() File "/usr/local/lib/python3.12/dist-packages/tensorflow/python/platform/self_check.py", line 63, in preload_check from tensorflow.python.platform import _pywrap_cpu_feature_guard ImportError: libnuma.so.1: cannot open shared object file: No such file or directory
pytorch-examples-wlang-gru	🔴	FAILED: MIGraphX is not within tolerance - check verbose output
pytorch-examples-wlang-lstm	🔴	FAILED: MIGraphX is not within tolerance - check verbose output
dlrm-criteoterabyte	✅	PASSED: MIGraphX meets tolerance
agentmodel	✅	PASSED: MIGraphX meets tolerance
unet	✅	PASSED: MIGraphX meets tolerance
resnet50v1	✅	PASSED: MIGraphX meets tolerance
bert_base_cased_fp16	✅	PASSED: MIGraphX meets tolerance
bert_large_uncased_fp16	🔴	FAILED: MIGraphX is not within tolerance - check verbose output
bert_large	✅	PASSED: MIGraphX meets tolerance
yolov5s	✅	PASSED: MIGraphX meets tolerance
tinyllama	✅	PASSED: MIGraphX meets tolerance
vicuna-fastchat	✅	PASSED: MIGraphX meets tolerance
whisper-tiny-encoder	✅	PASSED: MIGraphX meets tolerance
whisper-tiny-decoder	✅	PASSED: MIGraphX meets tolerance
distilgpt2_fp16	🔴	FAILED: MIGraphX is not within tolerance - check verbose output
llama2_7b	✅	PASSED: MIGraphX meets tolerance
qwen1.5-7b	✅	PASSED: MIGraphX meets tolerance
phi3-3.8b	✅	PASSED: MIGraphX meets tolerance
llama3-8b	✅	PASSED: MIGraphX meets tolerance
whisper-large-encoder	✅	PASSED: MIGraphX meets tolerance
whisper-large-decoder	✅	PASSED: MIGraphX meets tolerance
mistral-7b	✅	PASSED: MIGraphX meets tolerance
FLUX.1-schnell	✅	PASSED: MIGraphX meets tolerance

[pnikolic-amd]

Can this be a cmake option instead?

@pfultz2 We want to have option to change behavior in with api option so it can be modified for specific environment in runtime. We don't want to have specific build.

[pfultz2]

We want to have option to change behavior in with api option so it can be modified for specific environment in runtime.

What does "specific environment" mean here? This still sounds global which means we dont need a backend option, we could use an env variable or just delete the hiprtc driver for that environment.