Restructuring of output directories/ directories on server

Our directory structure for our release artifacts is overly complicated (probably due to organic growth).

On an average (mirror) server we have the top-level directory which tracks our package repositories. Alongside that we have:

• an iso directory, holding the currently available releases (i.e. directories in the date format YYYY.MM.DD) of our .iso and bootstrap images and the netboot artifacts (soon in the scope of the releng project)
  • on the level of the per-release directory, we currently have the archboot directory, containing its own per-release directories
• an images directory, holding the currently available releases of arch-boxes (i.e. directories in the format vYYYYMMDD.build_number)

While it is unlikely that we will ever have e.g. a package repository called iso or images, we are polluting the global space of that directory with our release artifact directories, which are on top of that not really uniquely named on a lower level.

I propose to move all release artifacts to a top-level releases directory, which will then hold a directory for each artifact type instead:

• archboot: for artifact releases of the archboot project
• images (or boxes?): for releases of the arch-boxes project
• netboot: for netboot artifact releases of the releng project
• iso: for iso artifact releases of the releng project
• bootstrap: for bootstrap artifact releases of the releng project
• ipxe: for iPXE artifact releases of the releng project

Eventually, this will lead us to a per-type directory layout, that is predictable:

releases
└── type
    ├── latest -> type-version
    ├── type-version.json
    ├── type-version.torrent
    └── type-version
        ├── type-version.ext
        ├── type-version.ext.checksum
        └── type-version.ext.sig

3 directories, 3 files

Byproducts

While looking into creating .torrent files for our releases (#5 (closed)) and how to approach e.g. FS#67203 I realized that the current directory layout will prevent us from having torrent files be created for directories (i.e. subsets of the files in a directory such as build artifacts and their signatures) and be able to use the mirror servers as web seeds for them.

With the proposed changes a set of artifacts in a per-release directory can be downloaded using a .torrent file, without conflicting with any other files in the download directory (e.g. a directory named <date> is not unique enough).

Changes

Ordering/moving the directory structure will require the following (known) changes:

arch-boxes

Changes to arch-boxes would entail to deploy the files to the different directory structure and to rename the per-release directories to something unique, such as boxes-<version>. A latest symlink should point to the directory with the current release (as is now the case).

archboot

The only change for archboot would be to deploy the artifacts to the new directory and rename the per-release directories to something unique, such as archboot-<version>. A latest symlink should point to the directory with the current release (as is now the case). New layout achieved for releases directory:

New layout for releases as follows archboot-release.sh:

archboot
├── <release> (e.g. 2021.09)
│   ├── boot
│   │   ├── vmlinuz_archboot_x86_64
│   │   ├── initramfs_archboot_x86_64
│   │   ├── amd-ucode.img
│   │   ├── intel-ucode.img
│   │   ├── sha256sum.txt
│   │   ├── licenses
│   │   │   ├── amd-ucode
│   │   │   │   └── LICENSE.amd-ucode
│   │   │   ├── intel-ucode
│   │   │   │   └── LICENSE
├───├── archlinux-archboot-20210916-1600-x86_64.iso
├───├── sha256sum.txt
├── latest (symlink to latest release)

archivetools

Archivetools is used to rsync the ISOs to places. The directory structure/ location needs to be adapted.

archlinux-repro

The project is affected because it uses the bootstrap images:

• https://github.com/archlinux/archlinux-repro/issues/97

infrastructure

The install_arch role needs to be adapted, after the change to the directory structure has been introduced (tracked in infrastructure#343).

releng

When looking at the default output of the current releng pipeline, it models after the status quo (apart from some new/ additional files and deployment of iPXE binaries, minus the .torrent file and manual signatures):

Current directory layout for releng artifacts

releng/output
├── arch
│   ├── boot
│   │   ├── amd-ucode.img
│   │   ├── amd-ucode.img.ipxe.sig
│   │   ├── intel-ucode.img
│   │   ├── intel-ucode.img.ipxe.sig
│   │   ├── licenses
│   │   │   ├── amd-ucode
│   │   │   │   └── LICENSE.amd-ucode
│   │   │   ├── intel-ucode
│   │   │   │   └── LICENSE
│   │   │   └── memtest86+
│   │   │       └── license.txt
│   │   ├── memtest
│   │   └── x86_64
│   │       ├── initramfs-linux.img
│   │       ├── initramfs-linux.img.ipxe.sig
│   │       ├── vmlinuz-linux
│   │       └── vmlinuz-linux.ipxe.sig
│   ├── grubenv
│   ├── pkglist.x86_64.txt
│   ├── version
│   └── x86_64
│       ├── airootfs.sfs
│       ├── airootfs.sfs.sig
│       └── airootfs.sha512
├── archlinux-2021.05.16-x86_64.iso
├── archlinux-2021.05.16-x86_64.iso.b2
├── archlinux-2021.05.16-x86_64.iso.md5
├── archlinux-2021.05.16-x86_64.iso.sha1
├── archlinux-2021.05.16-x86_64.iso.sha256
├── archlinux-2021.05.16-x86_64.iso.sha512
├── archlinux-2021.05.16-x86_64.iso.zsync
├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz
├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.b2
├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.md5
├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.sha1
├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.sha256
├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.sha512
├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.zsync
├── ipxe
│   ├── ipxe-arch.efi
│   ├── ipxe-arch.efi.b2
│   ├── ipxe-arch.efi.md5
│   ├── ipxe-arch.efi.sha1
│   ├── ipxe-arch.efi.sha256
│   ├── ipxe-arch.efi.sha512
│   ├── ipxe-arch.lkrn
│   ├── ipxe-arch.lkrn.b2
│   ├── ipxe-arch.lkrn.md5
│   ├── ipxe-arch.lkrn.sha1
│   ├── ipxe-arch.lkrn.sha256
│   ├── ipxe-arch.lkrn.sha512
│   ├── ipxe-arch.pxe
│   ├── ipxe-arch.pxe.b2
│   ├── ipxe-arch.pxe.md5
│   ├── ipxe-arch.pxe.sha1
│   ├── ipxe-arch.pxe.sha256
│   └── ipxe-arch.pxe.sha512
└── metrics.txt

9 directories, 51 files

I propose the following changes, so that we may have uniquely named directories (for the releng build artifacts):

New directory layout for releng artifacts

releng/output
├── bootstrap-2021.05.16
│   ├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz
│   ├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.b2
│   ├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.md5
│   ├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.sha1
│   ├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.sha256
│   ├── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.sha512
│   └── archlinux-bootstrap-2021.05.16-x86_64.tar.gz.zsync
├── ipxe-2021.05.16
│   ├── ipxe-arch.efi
│   ├── ipxe-arch.efi.b2
│   ├── ipxe-arch.efi.md5
│   ├── ipxe-arch.efi.sha1
│   ├── ipxe-arch.efi.sha256
│   ├── ipxe-arch.efi.sha512
│   ├── ipxe-arch.lkrn
│   ├── ipxe-arch.lkrn.b2
│   ├── ipxe-arch.lkrn.md5
│   ├── ipxe-arch.lkrn.sha1
│   ├── ipxe-arch.lkrn.sha256
│   ├── ipxe-arch.lkrn.sha512
│   ├── ipxe-arch.pxe
│   ├── ipxe-arch.pxe.b2
│   ├── ipxe-arch.pxe.md5
│   ├── ipxe-arch.pxe.sha1
│   ├── ipxe-arch.pxe.sha256
│   └── ipxe-arch.pxe.sha512
├── iso-2021.05.16
│   ├── archlinux-2021.05.16-x86_64.iso
│   ├── archlinux-2021.05.16-x86_64.iso.b2
│   ├── archlinux-2021.05.16-x86_64.iso.md5
│   ├── archlinux-2021.05.16-x86_64.iso.sha1
│   ├── archlinux-2021.05.16-x86_64.iso.sha256
│   ├── archlinux-2021.05.16-x86_64.iso.sha512
│   └── archlinux-2021.05.16-x86_64.iso.zsync
├── metrics.txt
└── netboot-2021.05.16
    ├── boot
    │   ├── amd-ucode.img
    │   ├── amd-ucode.img.ipxe.sig
    │   ├── intel-ucode.img
    │   ├── intel-ucode.img.ipxe.sig
    │   ├── licenses
    │   │   ├── amd-ucode
    │   │   │   └── LICENSE.amd-ucode
    │   │   ├── intel-ucode
    │   │   │   └── LICENSE
    │   │   └── memtest86+
    │   │       └── license.txt
    │   ├── memtest
    │   └── x86_64
    │       ├── initramfs-linux.img
    │       ├── initramfs-linux.img.ipxe.sig
    │       ├── vmlinuz-linux
    │       └── vmlinuz-linux.ipxe.sig
    ├── grubenv
    ├── pkglist.x86_64.txt
    ├── version
    └── x86_64
        ├── airootfs.sfs
        ├── airootfs.sfs.sig
        └── airootfs.sha512

11 directories, 51 files

The above should live in the respective artifact specific directories (e.g. bootstrap, netboot, iso, ipxe) below the releases directory. A latest symlink should point to the directory with the current release per release artifact type (as is now the case for the entire combined directory).

Deprecation of current directory layout

The switch to the new directory layout should not remove the previous directory layout directly. The release artifacts in the previous locations need to be cycled out of existence on a monthly basis (this gives users a grace period of about three months after the change).

After a three month grace period the top-level images and iso directories will be removed.

To deprecate our current directory layout, we will need to find all internal projects, that make use of e.g. the bootstrap or iso images and introduce the necessary changes to them, after switching to the new layout. Tickets towards these projects should be linked to this issue and be "blocked by" it.

Announcements

The above changes require up-front communication with our own projects (e.g. the auto-generated script (#9) needs to be updated to point to the correct files, the generation of the torrent file(s) needs to be adapted), mirror administrators and users.

• News entry
• arch-mirrors
• (optional) arch-general