Commit 85d243ff authored by Gerardo Exequiel Pozzi's avatar Gerardo Exequiel Pozzi
Browse files

[archiso] Use dm-snapshot instead of aufs2 (A.K.A. "The Big Commit")



* Use device mapper + snapshot module, instead union layer filesystem.
  * A block-level approach vs vfs-level.
  * No more unofficial (Linux) things.
  * More memory is needed.
* Refactor mkarchiso.
* Refactor hooks/archiso.
* Fix install/archiso_pxe_nbd
  (due recent change in mkinitcpio-0.6.15 on checked_modules()/all_modules())
  [Thanks Dave for the improved workaround]
* New configs/releng to build official images.
  * Works with a Bash script instead of Makefile.
    (better control and easy to maintain)
* Remove configs/syslinux-iso.
* Remove archiso2dual script. Integrate functionality in configs/releng.
* New configs/baseline to build the most basic live medium or use as template.
* New README (draft). [Thanks Dieter for fixing english grammar]
Signed-off-by: default avatarGerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar>
parent 4a1bd4c7
Archiso For Dummies Like Me and You
-------------------------------------
- What the hell is Archiso?
Archiso is a small set of bash scripts that is capable of building fully
functional Arch Linux based live CDs. It is a very generic tool, so it
could potentially be used to generate anything from rescue systems,
to install disks, to special interest live CD systems, and who knows what
else. Simply put, if it involves Arch on a shiny coaster, it can do it.
- Alright, so how does one install it?
First off, Archiso has some dependencies:
- mkinitcpio
- cdrkit
- squashfs-tools
- aufs2 (only needed in target media)
- aufs2-util (only needed in target media)
- devtools for mkarchroot
- syslinux
- nbd
- mkinitcpio-nfs-utils
Archiso itself can be installed with the handy dandy included Makefile,
and the incantation 'make install'.
- Great, so how do you use this thing?
The heart and soul of Archiso is mkarchiso. All of its options are
documented in its usage output, so we won't go into detail here.
Instead, let's go over the general process.
The first thing you should probably do is create a directory to work
in, and cd to it. This'll help keep things organized. Next, you'll want
to create a mkinitcpio config file that suits your needs. Typically this
means modifying whatever hooks you want. A typical set of hooks for
archiso looks something like this:
HOOKS="base udev archiso archiso_pxe_nbd pata scsi sata usb fw pcmcia filesystems usbinput"
It's probably worth mentioning that hardware autodetection and things
of that nature do not belong here. Only what's necessary to get the system
on its feet, and out of the initcpio really belong here, fancier stuff
can be done on the booted system anyway.
You'll also want to create a list of packages you want installed on your
live CD system. A file full of package names, one-per-line, is the format
for this. Typically you'll want BASE and a kernel as a bare minimum, but
you're free to install whatever else you want. This is *great* for
special interest live CDs, just specify packages you want and gogogo.
The last item of importance is what are called addons. Basically this
means any other crap you might want to include on your live CD, including
binary package repos, special configurations, random files, we don't
know, be creative. mkarchiso expects them all to be put in a single
directory, with an fstab-like config file. Currently two types of addons
are supported, squashfs images that get layered onto the root union, and
plain directories which can be bind mounted anywhere under the root.
If you want to add a squashfs union layer:
- Set up whatever you want to include in a separate directory someplace,
as if that directory was / . Then run mksquahfs on it, and copy the
created image to your addons directory.
- Add an entry to your addons config file (which must be named 'config',
by the way). Typical squashfs entries look like this:
live_overlay.sqfs / squashfs
Where the first component is the path to the image relative to your
addons directory, the second is the mountpoint (irrelevant for
squashfs, they will all get layered at /) and of course the third
component is the type.
- Be aware that the order of entries on the config matters! Entries will
be layered on top of one another, later entries are mounted _UNDER_
earlier entries (an unfortunate counterintuitive result of the way we
have to mount the unions).
If you want to add plain directories to bind mount:
- Set up your directory somewhere, and copy it to your addon directory.
- Add an entry to your addons config file, example entry:
core /packages bind
where the first component is the path to the directory relative to
your addons directory, the second component is where you'd like it
bind-mounted relative to the live CD's root filesystem, and the last
component is the type.
- How can build installation mediums like provided by Arch Linux?
- Just follow these next steps as root.
- Note that mkarchroot is optional, but with it, will ensure to have
a clean enviroment for building isos.
- This clean chroot, will take about 400MB (+130MB with all needed tools).
- After make, max space usage is about 2GB.
- In last step instead of just execute make, can be more selective:
Execute "make net-iso" or make "core-iso".
Do not execute make net-iso after make core-iso, otherwise net-iso
will be really a core-iso.
pacman -S devtools --needed
mkarchroot /tmp/somedir base
mkarchroot -r bash /tmp/somedir
# vi/nano /etc/pacman.d/mirrorlist and uncomment your prefered mirror.
pacman -S git squashfs-tools syslinux devtools cdrkit make nbd mkinitcpio-nfs-utils
cd /tmp
git clone git://projects.archlinux.org/archiso.git
cd archiso/archiso
make install
cd ../configs/syslinux-iso/
make
Done!
vim: textwidth=72
INDEX
-----
* Image types generated by mkarchiso.
* File format for aitab.
* Why the /isolinux and /arch/boot/syslinux directories?
* Building the most basic Arch Linux live media. (configs/baseline)
* Building official Arch Linux live media. (configs/releng)
*** Image types generated by mkarchiso.
* image-name.sfs SquashFS image with all files directly on it.
* image-name.fs.sfs SquashFS with only one file inside (image-name.fs),
which is an image of some type of filesystem
(ext4, ext3, ext2, xfs), all files reside on it.
* image-name.fs Like image-name.fs.sfs but without SquashFS.
(For testing purposes only. The option copytoram
for archiso hook does not have any effect on these images)
*** File format for aitab.
The aitab file holds information about the filesystems images that must be
created by mkarchiso and mounted at initramfs stage from the archiso hook.
It consists of some fields which define the behaviour of images.
# <img> <mnt> <arch> <sfs_comp> <fs_type> <fs_size>
<img> Image name without extension (.fs .fs.sfs .sfs).
<mnt> Mount point.
<arch> Architecture { i686 | x86_64 | any }.
<sfs_comp> SquashFS compression type { gzip | lzo | xz }.
A special value of "none" denotes no usage of SquashFS.
<fs_type> Set the filesystem type of the image { ext4 | ext3 | ext2 | xfs }.
A special value of "none" denotes no usage of a filesystem.
In that case all files are pushed directly to SquashFS filesystem.
<fs_size> An absolute value of file system image size in MiB.
(example: 100, 1000, 4096, etc)
A relative value of file system free space [in percent].
{1%..99%} (example 50%, 10%, 7%).
This is an estimation, and calculated in a simple way.
Space used + 10% (estimated for metadata overhead) + desired %
Note: Some combinations are invalid, example: sfs_comp=none and fs_type=none
*** Why the /isolinux and /arch/boot/syslinux directories?
The /isolinux directory holds files needed for the ISOLINUX boot loader
module of SYSLINUX. ISOLINUX can not find config files on
/arch/boot/syslinux, like other boot loaders modules (EXTLINUX, SYSLINUX, etc).
When make your custom boot-pendrive, you need to copy /arch directory to it.
/isolinux/isolinux.cfg just holds /arch/boot/syslinux/syslinux.cfg
*** Building the most basic Arch Linux live media. (configs/baseline)
* First install devtools if needed, mkarchroot needs it.
[host] # pacman -S devtools
* Create a chroot to work on it.
(prefix with linux32 if you want to build a 32 bits enviroment under 64 bits)
[host] # mkarchroot /tmp/chroot base
* Enter it. (prefix with linux32 if needed).
[host] # mkarchroot -r bash /tmp/chroot
* Create a loopback device.
(mkarchiso does not create it, use other number if not available)
[chroot] # mknod /dev/loop0 b 7 0
* Setup a mirror.
[chroot] # echo 'Server = MIRROR/archlinux/$repo/os/$arch' >> /etc/pacman.d/mirrorlist
* Install aditional packages needed for mkarchiso.
(git is only needed to get a copy of archiso.git)
[chroot] # pacman -S git squashfs-tools syslinux devtools cdrkit make
* Install archiso.
[chroot] # cd /tmp
[chroot] # git clone git://projects.archlinux.org/archiso.git
[chroot] # cd archiso/archiso
[chroot] # make install-program
* Build a basic iso.
[chroot] # cd /tmp/archiso/configs/baseline
[chroot] # ./build.sh
* Exit from chroot.
[chroot] # exit
Note: If you want to customize, just see the configs/releng directory which is
used to build official images with much more things.
*** Building official Arch Linux live media. (configs/releng)
Note: These steps should be done with 64 bits support.
* Prepare a 32 bit chroot enviroment.
linux32 mkarchroot /tmp/chroot32 base
linux32 mkarchroot -r bash /tmp/chroot32
echo 'Server = MIRROR/archlinux/$repo/os/$arch' >> /etc/pacman.d/mirrorlist
pacman -S squashfs-tools syslinux devtools cdrkit make mkinitcpio-nfs-utils nbd --noconfirm --needed
exit
* Prepare a 64 bits chroot enviroment.
mkarchroot /tmp/chroot64 base
mkarchroot -r bash /tmp/chroot64
echo 'Server = MIRROR/archlinux/$repo/os/$arch' >> /etc/pacman.d/mirrorlist
pacman -S squashfs-tools syslinux devtools cdrkit make mkinitcpio-nfs-utils nbd --noconfirm --needed
exit
* Create a shared directory which archiso can access from both chroot enviroments.
mkdir /tmp/shared
cd /tmp/shared
git clone git://github.com/djgera/archiso.git -b dm-snapshot
cd
mount --bind /tmp/shared /tmp/chroot32/tmp
mount --bind /tmp/shared /tmp/chroot64/tmp
* Enter 32 bits chroot enviroment, install mkarchiso,
then build core and netinstall single images.
linux32 mkarchroot -r bash /tmp/chroot32
mknod /dev/loop1032 b 7 1032
cd /tmp/archiso/archiso
make install-program
cd ../configs/releng/
./build.sh all_iso_single
* Enter 64 bits chroot enviroment, install mkarchiso,
then build core and netinstall single images.
mkarchroot -r bash /tmp/chroot64
mknod /dev/loop1064 b 7 1064
cd /tmp/archiso/archiso
make install-program
cd ../configs/releng/
./build.sh all_iso_single
* Finally build core and netinstall dual images
from any of the chroot enviroments. (The final result is the same).
mkarchroot -r bash /tmp/chroot64
cd /tmp/archiso/configs/releng/
./build.sh all_iso_dual
Archiso For Dummies Like Me and You
-------------------------------------
- What the hell is Archiso?
Archiso is a small set of bash scripts that is capable of building fully
functional Arch Linux based live CDs. It is a very generic tool, so it
could potentially be used to generate anything from rescue systems,
to install disks, to special interest live CD systems, and who knows what
else. Simply put, if it involves Arch on a shiny coaster, it can do it.
- Alright, so how does one install it?
First off, Archiso has some dependencies:
- mkinitcpio
- cdrkit
- squashfs-tools
- aufs2 (only needed in target media)
- aufs2-util (only needed in target media)
- devtools for mkarchroot
- syslinux
- nbd
- mkinitcpio-nfs-utils
Archiso itself can be installed with the handy dandy included Makefile,
and the incantation 'make install'.
- Great, so how do you use this thing?
The heart and soul of Archiso is mkarchiso. All of its options are
documented in its usage output, so we won't go into detail here.
Instead, let's go over the general process.
The first thing you should probably do is create a directory to work
in, and cd to it. This'll help keep things organized. Next, you'll want
to create a mkinitcpio config file that suits your needs. Typically this
means modifying whatever hooks you want. A typical set of hooks for
archiso looks something like this:
HOOKS="base udev archiso pata scsi sata usb fw pcmcia filesystems usbinput"
It's probably worth mentioning that hardware autodetection and things
of that nature do not belong here. Only what's necessary to get the system
on its feet, and out of the initcpio really belong here, fancier stuff
can be done on the booted system anyway.
You'll also want to create a list of packages you want installed on your
live CD system. A file full of package names, one-per-line, is the format
for this. Typically you'll want BASE and a kernel as a bare minimum, but
you're free to install whatever else you want. This is *great* for
special interest live CDs, just specify packages you want and gogogo.
The last item of importance is what are called addons. Basically this
means any other crap you might want to include on your live CD, including
binary package repos, special configurations, random files, we don't
know, be creative. mkarchiso expects them all to be put in a single
directory, with an fstab-like config file. Currently two types of addons
are supported, squashfs images that get layered onto the root union, and
plain directories which can be bind mounted anywhere under the root.
If you want to add a squashfs union layer:
- Set up whatever you want to include in a separate directory someplace,
as if that directory was / . Then run mksquahfs on it, and copy the
created image to your addons directory.
- Add an entry to your addons config file (which must be named 'config',
by the way). Typical squashfs entries look like this:
live_overlay.sqfs / squashfs
Where the first component is the path to the image relative to your
addons directory, the second is the mountpoint (irrelevant for
squashfs, they will all get layered at /) and of course the third
component is the type.
- Be aware that the order of entries on the config matters! Entries will
be layered on top of one another, later entries are mounted _UNDER_
earlier entries (an unfortunate counterintuitive result of the way we
have to mount the unions).
If you want to add plain directories to bind mount:
- Set up your directory somewhere, and copy it to your addon directory.
- Add an entry to your addons config file, example entry:
core /packages bind
where the first component is the path to the directory relative to
your addons directory, the second component is where you'd like it
bind-mounted relative to the live CD's root filesystem, and the last
component is the type.
- How can build installation mediums like provided by Arch Linux?
- Just follow these next steps as root.
- Note that mkarchroot is optional, but with it, will ensure to have
a clean enviroment for building isos.
- This clean chroot, will take about 400MB (+130MB with all needed tools).
- After make, max space usage is about 2GB.
- In last step instead of just execute make, can be more selective:
Execute "make net-iso" or make "core-iso".
Do not execute make net-iso after make core-iso, otherwise net-iso
will be really a core-iso.
pacman -S devtools --needed
mkarchroot /tmp/somedir base
mkarchroot -r bash /tmp/somedir
# vi/nano /etc/pacman.d/mirrorlist and uncomment your prefered mirror.
pacman -S git squashfs-tools syslinux devtools cdrkit make nbd mkinitcpio-nfs-utils
cd /tmp
git clone git://projects.archlinux.org/archiso.git
cd archiso/archiso
make install
cd ../configs/syslinux-iso/
make
Done!
vim: textwidth=72
all:
install: all
install: install-program install-examples install-doc
install-program:
# install to sbin since script only usable by root
install -D -m 755 mkarchiso $(DESTDIR)/usr/sbin/mkarchiso
# testiso can be used by anyone
......@@ -12,11 +14,16 @@ install: all
install -D -m 644 install/archiso_pxe_nbd $(DESTDIR)/lib/initcpio/install/archiso_pxe_nbd
install -D -m 644 hooks/archiso_loop_mnt $(DESTDIR)/lib/initcpio/hooks/archiso_loop_mnt
install -D -m 644 install/archiso_loop_mnt $(DESTDIR)/lib/initcpio/install/archiso_loop_mnt
# install docs and examples
install-examples:
# install examples
install -d -m 755 $(DESTDIR)/usr/share/archiso/
cp -r ../configs $(DESTDIR)/usr/share/archiso/configs
install -D -m 644 ../README $(DESTDIR)/usr/share/doc/archiso/README
install-doc:
install -d -m 755 $(DESTDIR)/usr/share/archiso/
install -D -m 644 ../README $(DESTDIR)/usr/share/doc/archiso/README
uninstall:
rm -f $(DESTDIR)/usr/sbin/mkarchiso
rm -f $(DESTDIR)/usr/bin/testiso
......@@ -25,4 +32,5 @@ uninstall:
rm -f $(DESTDIR)/lib/initcpio/hooks/archiso_pxe_nbd
rm -f $(DESTDIR)/lib/initcpio/install/archiso_pxe_nbd
rm -rf $(DESTDIR)/usr/share/archiso/
rm -rf $(DESTDIR)/usr/share/doc/archiso/
.PHONY: install install-program install-examples install-doc uninstall
# args: source, mountpoint
_mnt_aufs() {
src="${1}"
mnt="${2}"
msg "::: Adding new aufs branch: ${src} to ${mnt}"
mkdir -p "${mnt}"
/bin/mount -t aufs -o remount,append:"${src}"=ro none "${mnt}"
# Initialize loopback device logic (we using on-demand mode)
# args: none
_init_loop_dev() {
loop_dev_cnt=99
}
# Call this function before _make_loop_dev() each time.
# args: none
_next_loop_dev() {
loop_dev_cnt=$((loop_dev_cnt+1))
}
# Setup a loopback device for image passed as arguemnt and echo the path to loopback device used.
# args: /path/to/image_file
_make_loop_dev() {
local img="${1}"
mknod /dev/loop${loop_dev_cnt} b 7 ${loop_dev_cnt} &> /dev/null
losetup /dev/loop${loop_dev_cnt} "${img}" &> /dev/null
echo /dev/loop${loop_dev_cnt}
}
# args: source, mountpoint
_mnt_bind() {
src="${1}"
mnt="${2}"
msg "::: Binding ${src} to ${mnt}"
_mnt_fs() {
local img="${1}"
local mnt="${2}"
local img_fullname="${img##*/}";
local img_name="${img_fullname%%.*}"
local ro_dev ro_dev_size ro_dev_fs_type rw_dev
mkdir -p "${mnt}"
/bin/mount -o bind "${src}" "${mnt}"
_next_loop_dev
ro_dev=$(_make_loop_dev "${img}")
ro_dev_size=$(blockdev --getsz ${ro_dev})
ro_dev_fs_type=$(blkid -o value -s TYPE -p ${ro_dev} 2> /dev/null)
dd of="/cowspace/${img_name}.cow" count=0 seek=${ro_dev_size} &> /dev/null
_next_loop_dev
rw_dev=$(_make_loop_dev "/cowspace/${img_name}.cow")
echo "0 ${ro_dev_size} snapshot ${ro_dev} ${rw_dev} N 8" | dmsetup create ${img_name}
msg ":: Mounting '/dev/mapper/${img_name}' (${ro_dev_fs_type}) to '${mnt}'"
if ! mount -t "${ro_dev_fs_type}" "/dev/mapper/${img_name}" "${mnt}" ; then
echo "ERROR: while mounting '/dev/mapper/${img_name}' to '${mnt}'"
launch_interactive_shell
fi
}
# args: /path/to/image_file, mountpoint
_mnt_squashfs() {
img="${1}"
mnt="${2}"
img_fullname="${img##*/}";
img_name="${img_fullname%.*}"
tmp_mnt="/ro_branch/${img_name}"
if [ "${copytoram}" = "y" ]; then
_mnt_sfs() {
local img="${1}"
local mnt="${2}"
local img_fullname="${img##*/}";
mkdir -p "${mnt}"
if [[ "${copytoram}" == "y" ]]; then
msg -n ":: Copying squashfs image to RAM..."
/bin/cp "${img}" "/copytoram/${img_fullname}"
if [ $? -ne 0 ]; then
echo "ERROR: while copy ${img} to /copytoram/${img_fullname}"
if ! cp "${img}" "/copytoram/${img_fullname}" ; then
echo "ERROR: while copy '${img}' to '/copytoram/${img_fullname}'"
launch_interactive_shell
fi
img="/copytoram/${img_fullname}"
msg "done."
fi
mkdir -p "${tmp_mnt}"
/bin/mount -r -t squashfs "${img}" "${tmp_mnt}"
if [ $? -ne 0 ]; then
echo "ERROR: while mounting ${img} to ${tmp_mnt}"
_next_loop_dev
msg ":: Mounting '${img}' (SquashFS) to '${mnt}'"
if ! mount -r -t squashfs $(_make_loop_dev "${img}") "${mnt}" &> /dev/null ; then
echo "ERROR: while mounting '${img}' to '${mnt}'"
launch_interactive_shell
fi
if [ "/${mnt#/*/}" = "/" ]; then
_mnt_aufs "${tmp_mnt}" "${mnt}"
else
_mnt_bind "${tmp_mnt}" "${mnt}"
fi
}
run_hook() {
if [ "x${arch}" = "x" ]; then
arch="$(uname -m)"
fi
if [ "x${rw_branch_size}" = "x" ]; then
rw_branch_size="75%"
fi
if [ "x${copytoram_size}" = "x" ]; then
copytoram_size="75%"
fi
if [ "x${archisobasedir}" = "x" ]; then
archisobasedir="arch"
fi
if [ "x${isomounts}" != "x" ]; then
isomounts="/bootmnt/${isomounts}"
[[ -z "${arch}" ]] && arch="$(uname -m)"
[[ -z "${cowspace_size}" ]] && cowspace_size="75%"
[[ -z "${copytoram_size}" ]] && copytoram_size="75%"
[[ -z "${archisobasedir}" ]] && archisobasedir="arch"
[[ -z "${archisodevice}" ]] && archisodevice="/dev/disk/by-label/${archisolabel}"
if [[ -z "${aitab}" ]]; then
aitab="/bootmnt/${archisobasedir}/aitab"
else
isomounts="/bootmnt/${archisobasedir}/isomounts"
aitab="/bootmnt/${aitab}"
fi
if [ "x${archisodevice}" = "x" ]; then
archisodevice="/dev/disk/by-label/${archisolabel}"
fi
# set mount handler for archiso
mount_handler="archiso_mount_handler"
}
# This function is called normally from init script, but it can be called
# as chain from other mount handlers.
# args: /path/to/newroot
archiso_mount_handler() {
newroot="${1}"
local newroot="${1}"
local fstype fserror
_init_loop_dev
msg ":: Waiting for boot device..."
while ! poll_device ${archisodevice} 30; do
while ! poll_device "${archisodevice}" 30; do
echo "ERROR: boot device didn't show up after 30 seconds..."
echo " Falling back to interactive prompt"
echo " You can try to fix the problem manually, log out when you are finished"
launch_interactive_shell
done
FSTYPE=$(blkid -o value -s TYPE -p ${archisodevice} 2> /dev/null)
if [ -n "${FSTYPE}" ]; then
if mount -r -t "${FSTYPE}" ${archisodevice} /bootmnt > /dev/null 2>&1; then
if [ -e "${isomounts}" ]; then
echo "SUCCESS: Mounted archiso volume successfully."
fserror="0"
fstype=$(blkid -o value -s TYPE -p "${archisodevice}" 2> /dev/null)
if [[ -n "${fstype}" ]]; then
if mount -r -t "${fstype}" "${archisodevice}" /bootmnt; then
if [[ -f "${aitab}" ]]; then
msg ":: Mounted archiso volume successfully."
fserror=0
else
echo "ERROR: Mounting was successful, but the ${isomounts} file does not exist."
fserror="1"
echo "ERROR: Mounting was successful, but the '${aitab}' file does not exist."
fserror=1
fi
else
echo "ERROR; Failed to mount ${archisodevice} (FS is ${FSTYPE})"
fserror="1"
echo "ERROR; Failed to mount '${archisodevice}' (FS is ${fstype})"
fserror=1
fi
else
echo "ERROR: ${archisodevice} found, but the filesystem type is unknown."
fserror="1"
echo "ERROR: '${archisodevice}' found, but the filesystem type is unknown."
fserror=1
fi
if [ "${fserror}" = "1" ]; then
if [[ ${fserror} -eq 1 ]]; then
echo " Falling back to interactive prompt"
echo " You can try to fix the problem manually, log out when you are finished"
launch_interactive_shell
fi
if [ "${copytoram}" = "y" ]; then
if [[ "${copytoram}" == "y" ]]; then
msg -n ":: Mounting /copytoram (tmpfs) filesystem, size=${copytoram_size}..."
mount -t tmpfs -o "size=${copytoram_size}",mode=0755 copytoram /copytoram
msg "done."
fi
msg -n ":: Mounting rw_branch (tmpfs) filesystem, size=${rw_branch_size}..."
mount -t tmpfs -o "size=${rw_branch_size}",mode=0755 rw_branch /rw_branch
msg -n ":: Mounting /cowspace (tmpfs) filesystem, size=${cowspace_size}..."