Use of system libraries

Description:

• pybind11 is in the packages depends array but USE_SYSTEM_PYBIND11=ON is not set and the default is off.
• eigen is also packaged but in the depends array and USE_SYSTEM_EIGEN_INSTALL is not set and the default is off.
• python-onnx again is packaged but not a dependency. This fails during cmake detection of the system library. I have not had time to investigate further.

Would you accept a merge request enabling these system libraries?

Additional info:

• python-pytorch 2.2.2-1

added priority3-normal severity4-low statusunconfirmed labels

assigned to @freswa, @toolybird, and @tpkessler

MRs are always welcome. Thank you, @loqs!

I am trying. Out of memory has killed my attempts so far. Are there any flags apart from HIPCC_COMPILE_FLAGS_APPEND and HIPCC_LINK_FLAGS_APPEND to reduce the number of jobs being used?

Feel free to provide me the diff and I can compile and upload it for you

I uploaded rel-3 to staging. Please check it out. The configure script said it used system deps.

That is what I was aiming for. Thank you.

I realized onnx is not detected when I incldued python-onnx as it does not ship a stand alone C++ library. I was able to get pytorch detecting onnx with the following (I know this is out of scope I believe every dependency Arch packages is now being used)

diff --git a/PKGBUILD b/PKGBUILD
index a95fcb6..7513f20 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -1,7 +1,8 @@
 # Maintainer: Torsten Keßler <tpkessler@archlinux.org>
 # Contributor: Butui Hu <hot123tea123@gmail.com>
 
-pkgname=python-onnx
+pkgbase=onnx
+pkgname=(onnx python-onnx)
 pkgver=1.16.0
 pkgrel=3
 epoch=1
@@ -9,18 +10,16 @@ pkgdesc='Open standard for machine learning interoperability'
 arch=('x86_64')
 url='https://onnx.ai'
 license=('MIT')
-depends=(
-  python
-  protobuf
-  python-protobuf
-  python-numpy
-)
 makedepends=(
   cmake
   git
+  protobuf
   pybind11
+  python
   python-build
   python-installer
+  python-numpy
+  python-protobuf
   python-setuptools
   python-wheel
 )
@@ -29,19 +28,46 @@ sha512sums=('9d0d6ca4d0808e12ef1c10a38c2232ee6f0f4c996186c66d16819f72c379d0d708a
 
 
 prepare() {
-  cd "${pkgname}"
+  cd $pkgbase
   # Remove cmake and protobuf dependencies since we don't want to pull the
   # corresponding packages from PyPI
   sed -i 's/requires = .*/requires = ["setuptools"]/' pyproject.toml
 }
 
 build() {
-  cd "${pkgname}"
-  python -m build --wheel --no-isolation
+  local _cmake_args=(
+    -DCMAKE_BUILD_TYPE='None'
+    -DCMAKE_INSTALL_PREFIX='/usr'
+    -DCMAKE_POSITION_INDEPENDENT_CODE=ON
+    -DCMAKE_VERBOSE_MAKEFILE=ON
+    -DONNX_USE_PROTOBUF_SHARED_LIBS=ON
+    -DBUILD_SHARED_LIBS=ON
+    -DONNX_BUILD_TESTS=OFF
+    -DBUILD_ONNX_PYTHON=ON
+    -Wno-dev
+  )
+  cmake -B build -S $pkgbase "${_cmake_args[@]}"
+  cmake --build build
+  cd $pkgbase
+  CMAKE_ARGS="${_cmake_args[@]}" python -m build --wheel --no-isolation
 }
 
-package() {
-  cd "${pkgname}"
+package_onnx() {
+depends=(
+  protobuf
+)
+  DESTDIR="$pkgdir" cmake --install build
+  install -Dm644 $pkgbase/LICENSE -t "${pkgdir}/usr/share/licenses/${pkgname}"
+}
+
+package_python-onnx() {
+depends=(
+  python
+  protobuf
+  python-protobuf
+  python-numpy
+)
+  cd $pkgbase
   python -m installer --destdir="${pkgdir}" dist/*.whl
   install -Dm644 LICENSE -t "${pkgdir}/usr/share/licenses/${pkgname}"
 }

@loqs could you MR this to the onnx package?

Turns out you already did that, should have checked!

Out of memory has killed my attempts so far.

@loqs I build with export MAX_JOBS=32 to address this.

@greyltc Thank you for the tip on the variable MAX_JOBS. I believe 32 would not have any effect on the system I was experiencing the OOM on as that would match core count which I believe is the default set in pytorch/tools/setup_helpers/cmake.py when MAX_JOBS and USE_NINJA are not set.

added scopeenhancement statusconfirmed labels and removed statusunconfirmed label

assigned to @svenstaro and @kgizdov and unassigned @freswa and @toolybird

mentioned in commit f2eac825

Anything more to do here? Can we close?

All done with what is currently packaged the next unbundling step would be xnnpack which could also be used by onnx and its dependencies fp16 gloo psimd fxdiv pthreadpool cpuinfo.

I will look into packaging xnnpack soon and then maybe python-pytorch and onnx can use that.

At second glance, XNNPACK doesn't appear to have any releases or tags. Not sure what would be reasonable to package, then.

Fedora use the commit from the torch release it will be linked against. I think Debian may do the same as matches 51a9875.