OpenWRT/LEDE: System Boot Sequence

In this article I will try to summary and analyze the boot sequnece of OpenWRT system in details.

Hardware

ci40

Software

OpenWRT for ci40

You may find difference amound different OpenWRT version and LEDE. But I believe that it won’t affect the understanding.

Example boot log from ci40
[ 2.824545] VFS: Mounted root (ext4 filesystem) readonly on device 179:1.
[ 2.833446] Freeing unused kernel memory: 244K (84733000 - 84770000)
[ 3.006884] init: Console is alive
[ 3.011436] init: - watchdog -
[ 3.329383] init: - preinit -
[ 6.570976] mount_root: mounting /dev/root
[ 6.579281] EXT4-fs (mmcblk0p1): re-mounted. Opts: (null)
[ 6.596450] procd: - early -
[ 6.599817] procd: - watchdog -
[ 7.301153] procd: - ubus -
[ 7.362047] procd: - init -

1. Overview

The whole system boot process is firstly shown in a nutshell:

  1. Bootloader e.g., U-Boot:
    It configures low level hardwares, loads Linux kernel imag and device tree blob, finally jumps to Linux kernel image in the RAM with a kernel cmdline;
  2. Kernel –> Hareware:
    The Linux kernel will init hardwares for everything built ‘static’ in the kernel;

  3. Kernel –> Filesystem:
    The root filesystem (via root=, rootfstype= etc parameters in the kernel cmdline) will be mounted;

  4. Kernel –> Init Process
    At last kernel will start init process (PID 1);

  5. OpenWRT –> Preinit:
    Before the real procd runs, a small init process is started. This process has the job of early system init.

  6. OpenWRT –> Procd:
    Once preinit is complete the init process is done and will do an exec on the real procd. This will replace init as pid1 with an instance of procd running as the new pid 1. The watchdog file descriptor is not closed. Instead it is handed over to the new procd process. The debug_level will also be handed over to the new procd instance if it was set via command line or during preinit.

OpenWRT system starts actually from step 4. In OpenWRT system this init process is not a normal process of Linux but a shell scripts desinaged for OpenWRT. Additionally this init process is also the first part of preinit of OpenWRT. Therefore this article will start analysis of this speciall init process.

2. Preinit

2.1 /etc/preinit

OpenWRT has many patches for different kinds of hardwares and unique applications which do not exist in the mainline kernel. From such patches, which are generic patches for certain Linux kernel version and shared amount all the routers, are located in openwrt/target/linux/generic/patches-/. No matter which version of Linux you choose, you can always find a patch called 921-use_preinit_as_init.patch which injects the OpenWRT init process at the first place of init processes list. This patch is shown below:

--- a/init/main.c
+++ b/init/main.c
@@ -960,7 +960,8 @@ static int __ref kernel_init(void *unuse
panic("Requested init %s failed (error %d).",
execute_command, ret);
}
- if (!try_to_run_init_process("/sbin/init") ||
+ if (!try_to_run_init_process("/etc/preinit") ||
+ !try_to_run_init_process("/sbin/init") ||
!try_to_run_init_process("/etc/init") ||
!try_to_run_init_process("/bin/init") ||
!try_to_run_init_process("/bin/sh"))

It is obviously that the init process is located at /etc/ in the deivce filesystem or openwrt/package/base-files/files/etc/ of source tree. Next I will analyze this scripts in detail. As usual the source code of this preinit is firstly shown

#!/bin/sh
# Copyright (C) 2006-2016 OpenWrt.org
# Copyright (C) 2010 Vertical Communications

[ -z "$PREINIT" ] && exec /sbin/init

export PATH="%PATH%"

pi_ifname=
pi_ip=192.168.1.1
pi_broadcast=192.168.1.255
pi_netmask=255.255.255.0

fs_failsafe_ifname=
fs_failsafe_ip=192.168.1.1
fs_failsafe_broadcast=192.168.1.255
fs_failsafe_netmask=255.255.255.0

fs_failsafe_wait_timeout=2

pi_suppress_stderr="y"
pi_init_suppress_stderr="y"
pi_init_path="%PATH%"
pi_init_cmd="/sbin/init"

. /lib/functions.sh
. /lib/functions/preinit.sh
. /lib/functions/system.sh

boot_hook_init preinit_essential
boot_hook_init preinit_main
boot_hook_init failsafe
boot_hook_init initramfs
boot_hook_init preinit_mount_root

for pi_source_file in /lib/preinit/*; do
. $pi_source_file
done

boot_run_hook preinit_essential

pi_mount_skip_next=false
pi_jffs2_mount_success=false
pi_failsafe_net_message=false

boot_run_hook preinit_main

In this script, the first command is:

[ -z "$PREINIT" ] && exec /sbin/init

According to the process from kernel to this script, the environment parameter PREINIT is NOT defined, thus /sbin/init will be excuted. The /sbin/init is actually the first init process of OpenWRT system. This program is one of programs from procd package. Another well-known program is procd that will be introduced later.

2.3 /sbin/init

We don’t need to analze the whole souce code of /sbin/init. At first the main function could give us a good overview. The main function of /sbin/init is located at procd/initd/init.c file. We don’t need to analze the whole souce code to understand. In the following, a diagram will be demonstrated to show the execution order.

OpenWRT /sbin/init

  1. early() is the first function in init. It has four main tasks:
    • early_mounts(): mount /proc, /sysfs, /dev, /tmp;
    • early_env(): set PATH parameter with /usr/sbin:/sbin:/usr/bin:/bin;
    • initializes /dev/console;
    • print the first message: “Console is a alive” from init as shown above;
  2. cmdline() is the second function which reads the kernel boot command line from /proc/cmdline and parses init_debug parameter;
  3. watchdog_init() initializes watchdog (/dev/watchdog) and print the second message “- watchdog -” as shown above;
  4. fork() a new thread to let /sbin/kmodloader load device drivers regarding /etc/modules-boot.d/;
  5. uloop_init() initializes the uloop which is an event loop implementation. Later procd and sh /etc/preinit will be managed by uloop;
  6. preinit() has four main tasks:
    • prints the third message: “- preinit -” as shown above;
    • fork() a new thread to excute /sbin/procd program with parameter -h /etc/hotplug-preinit.json. A callback function called spawn_procd() will be excuted after /sbin/procd is finished.
      Note: spawn_procd() will read system debug level from /tmp/debuglevel and set it to env DBGLVL. It also sets watchdog fd to env WDTFD. At last it will fork the real /sbin/procd deamon.
    • setenv(“PREINIT”, “1”, 1);
    • fork() a new thread to excute sh /etc/preinit. This will be the second time to excute this init script.
      Note: child thread sh /etc/preinit will be added into uloop by uloop_process_add() together with a callbakc function as
plugd_proc.cb = plugd_proc_cb;

When sh /etc/preinit is finished, the callback function plugd_proc_cb()

static void
plugd_proc_cb(struct uloop_process *proc, int ret)
{
        proc->pid = 0;
}

will be excuted. It is clear that it will set the pid with 0 to show that sh /etc/preinit is finished.

  1. uloop_run();

2.2 /etc/preinit again

Now the init process comes to /etc/preinit again. The first part of this script is shown below:

export PATH="%PATH%"

pi_ifname=
pi_ip=192.168.1.1
pi_broadcast=192.168.1.255
pi_netmask=255.255.255.0

fs_failsafe_ifname=
fs_failsafe_ip=192.168.1.1
fs_failsafe_broadcast=192.168.1.255
fs_failsafe_netmask=255.255.255.0

fs_failsafe_wait_timeout=2

pi_suppress_stderr="y"
pi_init_suppress_stderr="y"
pi_init_path="%PATH%"
pi_init_cmd="/sbin/init"

These statements are simply variables definition.

Then it defined functions from following locations

. /lib/functions.sh
. /lib/functions/preinit.sh
. /lib/functions/system.sh

boot_hook_init is defined in /lib/functions/preinit.sh and used as

boot_hook_init preinit_essential
boot_hook_init preinit_main
boot_hook_init failsafe
boot_hook_init initramfs
boot_hook_init preinit_mount_root

to defined 5 hook nodes. In the following loop

for pi_source_file in /lib/preinit/*; do
. $pi_source_file
done

scripts located in /lib/preinit/ will be excuted. These scripts have defined functions which will be added to coresponding hook nodes by boot_hook_add.

At last /etc/preinit will excute boot_run_hook to find and excute functions hooked at certain node. In current system it will start from boot_run_hook preinit_main.

3. Watchdog

If the watchdog dev /dev/watchdog exists, setting watchdog timeout as 30 second. If the Linux kernel doese not receive any data, the system will be rebooted. The process use uloop to write some data into Linux kernel periodly (5s), which indicates that the process is woring well.

4. procd

LEDE uses procd for booting the system, managing processes and handling parts of the kernel/userland interaction. It can be considered similar to systemd on full blown distributions. Here is a list of tasks that procd will do for us

Procd will first do some basic process init such as setting itself to be owner of its own process group and setting up signals. We are now ready to bring up the userland in the following order

  • find out if a watchdog file descriptor was passed by the init process and start up the watchdog;
  • setup /dev/console to be our stdin/out/err;
  • start the coldplug process using the full rule set (/etc/hotplug.json). This is done by manually triggering all events that have already happened using udevtrigger;
  • start ubus, register it as a service and connect to it.
    The basic system bringup is now complete, procd is up and running and can start handling daemons and services.

procd has 6 states, STATE_EARLY, STATE_UBUS, STATE_INIT,STATE_RUNNING,STATE_SHUTDOWN, STATE_HALT. the procd state is changed from the first one until the last one. Current state is located in global variable state. The state can be changed by calling procd_state_next()

static void state_enter(void)
{
    char ubus_cmd[] = "/sbin/ubusd";

    switch (state) {
    case STATE_EARLY:
        LOG("- early -\n");
        watchdog_init(0);
        hotplug("/etc/hotplug.json");
        procd_coldplug();
        break;

    case STATE_UBUS:
        // try to reopen incase the wdt was not available before coldplug
        watchdog_init(0);
        set_stdio("console");
        LOG("- ubus -\n");
        procd_connect_ubus();
        service_start_early("ubus", ubus_cmd);
        break;

    case STATE_INIT:
        LOG("- init -\n");
        procd_inittab();
        procd_inittab_run("respawn");
        procd_inittab_run("askconsole");
        procd_inittab_run("askfirst");
        procd_inittab_run("sysinit");

        // switch to syslog log channel
        ulog_open(ULOG_SYSLOG, LOG_DAEMON, "procd");
        break;

    case STATE_RUNNING:
        LOG("- init complete -\n");
        procd_inittab_run("respawnlate");
        procd_inittab_run("askconsolelate");
        break;

    case STATE_SHUTDOWN:
        /* Redirect output to the console for the users' benefit */
        set_console();
        LOG("- shutdown -\n");
        procd_inittab_run("shutdown");
        sync();
        break;

    case STATE_HALT:
        // To prevent killed processes from interrupting the sleep
        signal(SIGCHLD, SIG_IGN);
        LOG("- SIGTERM processes -\n");
        kill(-1, SIGTERM);
        sync();
        sleep(1);
        LOG("- SIGKILL processes -\n");
        kill(-1, SIGKILL);
        sync();
        sleep(1);
#ifndef DISABLE_INIT
        if (reboot_event == RB_POWER_OFF)
            LOG("- power down -\n");
        else
            LOG("- reboot -\n");

        /* Allow time for last message to reach serial console, etc */
        sleep(1);

        /* We have to fork here, since the kernel calls do_exit(EXIT_SUCCESS)
         * in linux/kernel/sys.c, which can cause the machine to panic when
         * the init process exits... */
        if (!vfork( )) { /* child */
            reboot(reboot_event);
            _exit(EXIT_SUCCESS);
        }
        while (1)
            sleep(1);
#else
        exit(0);
#endif
        break;

    default:
        ERROR("Unhandled state %d\n", state);
        return;
    };
}

4.1 STATE_EARLY

In this state some preparartion work will be done before init.

  • init watchdog. Timeout is 30s;
  • hotplug(“/etc/hotplug.json”); will enable monitoring hotplug event according to the rules defined in /etc/hotplug.json;
  • in procd_coldplug function, /dev is mouted. udevtrigger will generate coldplug events for hotplug monitoring;
  • When udevstrigger is finished, its callback function procd_state_next() will change the state from STATE_EARLY to STATE_UBUS;

4.2 STATE_UBUS

  • watchdog_init(0); Init watchdoa again;
  • set_stdio(“console”); Setup standar IO device to console;
  • procd_connect_ubus(void); A timer is set to reconnect periodly with ubusd even ubusd may not exist yet. uloop_run() will run after init work finished. After procd connects ubusd, it will register service main_object, system_object and watch_object;
    • in ubus_connect_cb, /var/run/ubus.sock(UBUS_UNIX_SOCKET) will be used to commnunicate with ubus. The state will be changed to STATE_INIT after the connection is successully builded.
  • service_init(); Init AVL tree for services and validators;
  • service_start_early(“ubus”, ubus_cmd); Start ubusd service deamon;

4.3 STATE_INIT

In this state the real init work will be done

  • Add global link table actions from /etc/inittab;
  • Sequential loading respawn, askconsole, askfirst and sysinit commands;
  • procd_inittab_run() Callback functions of *respawn**, askconsole, askfirst and sysinit will be excuted. Regarding /etc/inittab, callback functions of sysinit and askfirst will be runned.
    • askfirst –> askfirst(): Start /sbin/askfirst to show Please press Enter to activate this console and then /bin/ash –login will be excuted.
    • sysinit –> runrc(): It wll call all the startup scripts which are located on /etc/rc.d. Actually it calls add_initd() to fork a process for every scripts, repectively, whichi is handled by struct runqueue. After that the callback function will call rcdone() to change the state from STATE_INIT to STATE_RUNNING.

4.4 STATE_RUNNING

In this state the procd will in the uloop_run() loop.

References
  1. https://wiki.openwrt.org/doc/techref/preinit_mount
  2. https://wiki.openwrt.org/doc/techref/requirements.boot.process
  3. http://trac.gateworks.com/wiki/OpenWrt/init
  4. https://www.lede-project.org/docs/procd.html
  5. http://ask.wrtnode.cc/question/43
  6. https://wiki.microduino.cc/index.php/%E7%AC%AC%E4%B8%80%E8%AF%BE–MicroWRT_%E7%B3%BB%E7%BB%9F%E5%90%AF%E5%8A%A8%E6%B5%81%E7%A8%8B#.E5.86.85.E6.A0.B8.E8.A1.A5.E4.B8.81
  7. http://lirobo.blogspot.de/2014/07/openwrt-boot.html
  8. http://www.cnblogs.com/rohens-hbg/p/5049085.html
  9. http://blog.leanote.com/post/shaokunyang@163.com/openwrt-cc-%E5%90%AF%E5%8A%A8%E6%B5%81%E7%A8%8B
  10. http://www.mamicode.com/info-detail-107649.html
  11. http://blog.csdn.net/wwx0715/article/details/41725917
  12. http://lirobo.blogspot.de/search/label/OpenWrt
  13. https://wiki.microduino.cn/index.php/MicroWRT_(OpenWRT%E5%85%BC%E5%AE%B9%E6%9D%BF%EF%BC%89%E6%95%99%E7%A8%8B
  14. http://see.sl088.com/wiki/Openwrt_%E5%90%AF%E5%8A%A8%E8%BF%87%E7%A8%8B
  15. https://segmentfault.com/a/1190000002392043
  16. http://blog.chinaunix.net/uid-26598889-id-3060545.html
  17. http://www.51hei.com/bbs/dpj-46073-1.html
  18. http://www.programdevelop.com/4531245/
Advertisements
This entry was posted in Ci40, LEDE, OpenWRT. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s