Mac OS X: Launchd in Depth

9j85m47547

Note: a old article topic, just for reference.

Launchd in Depth


Friday, July 08 2005 @ 04:52 pm MDT

Contributed by: macshome


Views: 140,263



When
Apple announced the UNIX based Mac OS X lots of sysadmins took notice.
For years the Linux crowd had been trying, and failing, to get UNIX to
the general desktop audience. I ran Linux for a while on Beige G3 and it
required a trip to Open Firmware and a bootconf loader on a floppy to
get it going at the time. When I installed my first Rhapsody build, all I
had to do was put the Caps Lock key on to get all the UNIXy goodness.
The times, they were a changing.


Fast forward several years and Apple is now the highest volume UNIX
vendor in the world. Furthermore the base of Mac OS X is open source and
Apple uses many OSS projects in Darwin. Apple also has created projects
and turned them out into the OSS world. (I'm well aware of the issues
that some in the community have with Apple's OSS participation level,
it's just not the focus of this article.) Recently Apple has loosed a
new beast onto the world and it's knocked many people for a loop.


Welcome, launchd. Seriously.

Read on for more...



Apple's position as the leading UNIX
vendor, and as a unified solution provider, gives them some unique
opportunities. One of these has been to take a look at some of the
traditional mishmashes of UNIX and try to sort them out a bit. Lookupd
is one of these efforts and it provides general resolver services for
everything from DNS to GID lookups on Mac OS X. Applications don't need
to know anything about how the machine is configured, they just ask
lookupd for the info. If you look at Mac OS X, and OpenStep before it,
you will see that the OS has been striving for this sort of
consolidation of services for a while now.


With Tiger, Apple has started to tackle the system of files and
processes that start a UNIX system. Before it was often hard to tell how
everything got cranked up. Was a daemon started by SystemStarter? Was
it started by xinetd? Was it started by RC? Was it started by cron? Was
it started by init or mach_init? Was it started by watchdog? Even worse,
what if you needed to add a service? It was fairly clear that
SystemStarter was what Apple wanted for startup items, but what about
launch on demand? Xientd seems obvious here, but not everything can be
launched by xinetd. What about service control? How do you easily start
and stop services so that the rest of the system is aware of them?


In all honesty, this is a problem that Microsoft had figured out a while
ago. There is nothing in Panther and earlier that compares to the
Services control panel, or MMC snap-in, on Windows. Mac OS X still
doesn't have that easy level of control, but the foundations are now
there with launchd and its buddy launchctl. (Incidentally, this is a
great opportunity for an industrious AppleScripter to whip out a Studio
app.)

Meet the team


When Apple decided to overhaul the 30 year old system for starting and
maintaining processes on UNIX it had to think big. launchd can do the
jobs of init, mach_init, xinetd, RC, SystemStarter, watchdog, and cron.
It is Apple's stated intention that they want to eliminate all of those
services in favor of launchd at some point in the future. There are two
main programs in the launchd system, launchd and launchctl. Briefly:


launchd manages the daemons at both a system and user level. Similar to
xinetd, launchd can start daemons on demand. Similar to watchdogd,
launchd can monitor daemons to make sure that they keep running. launchd
also has replaced init as PID 1 on Mac OS X and as a result it is
responsible for starting the system at boot time.


launchctl is our window into the world of launchd. Using launchctl we
can load and unload daemons, start and stop launchd controlled jobs, get
system utilization statistics for launchd and its child processes, and
set environment settings.


A third, and the most confusing, part of the system are the plist files
that define the services to be loaded into launchd with launchctl.
Stored in the LaunchAgents or LaunchDaemons  of any Library, the XML
files have around thirty different keys that can be set, and the syntax
is not very pretty. All of the options are laid out in the launchd.plist
man page, which hides them nicely by using an obscure name. Even after
reading the substantial man page you still have no real clear idea of
how to use the keys to construct a valid plist, and that's where
thieving from the default Apple plists comes in handy.


Time to dig in...

launchd


When it comes down to it, launchd has two main tasks. The first is to
boot the system and the second is to load and maintain services. Let's
take a look at the boot part of the job first.


When Mac OS X boots it goes through quite a few steps. Here is a very
simplified view of the 10.4 system startup.


1. The BootROM activates, does its thing to the hardware, and then loads BootX.

2. BootX loads the kernel, spins the gear, loads any needed kexts, and then the kernel loads launchd.

3. launchd then runs /etc/rc, scans through
/System/Library/LaunchDaemons and /Library/LaunchDaemons and acts on the
plists as needed, and finally starts the loginwindow.


Easy huh?


The first one we should look at here is step 2. This is a huge change
for Mac OS X, and *NIX in general, in that some flavor of init is not
loaded here. Indeed, launchd is now PID 1 on Mac OS X.


In step three, launchd scans through a few different directories for
jobs to run. There are two different folders types that are scanned. The
LaunchDaemons folders should contain items that will run as root,
generally background processes. The LaunchAgents folders should contain
jobs, called agent applications, that will run as a user or in the
context of userland. Often these can be scripts, other foreground items,
and they can even include a user interface. When we get to our example
we will be creating a user-specific LaunchAgent job. These directories
are all kept in the typical Libraries of Mac OS X.


launchd is very different from SystemStarter in that it may not actually
launch all the daemons at boot time. Key to launchd, and similar to
xinted, is the idea of launch on demand daemons. When launchd scans
through the job plists at boot time it reserves and listens on all of
the ports requested by those jobs. If so indicated in the plist by the
"OnDemand" key, the daemon is not actually loaded at the time. Rather
launchd will listen on the port, start the daemon when needed, and shut
it down when it is not. After a daemon is loaded, launchd will keep
track of it and make sure it is running if needed. In this way it is
like watchdogd, and shares watchdogd's requirement that processes do not
attempt to fork or daemonize on their own. If a process goes into the
background launchd will lose track of it and attempt to relaunch it.


This is why Tiger boots so fast. The system only has to register the
daemons that are to run, not actually launch them. In fact the progress
bar that appears is just a placebo that doesn't really show anything
other than the passage of time. This explains why the bar never reaches
the end before the login window appears sometimes. (For fun you can run
it any time you like by executing /usr/libexec/WaitingForLoginWindow.)
In addition to incoming requests, there are other ways to define launch
on demand and we will take a look at them in just a few moments.


The hardest part to manage during a launchd boot is dependancies.  
SystemStarter had a very simple system of dependancies that used the
"Uses", "Requires", and "Provides" keys in the plist of a startup item.
There are two main strategies when creating launch dependancies on 10.4.
You can use IPC which will allow the daemons to talk amongst themselves
to work it out or you can watch files or paths for changes. Using IPC
is much more subtle than the SystemStarter's keys and requires more work
from the developer, but it should lead to cleaner and quicker startups.
It does however, put dependancies out of the reach of many admins,
myself included, as we aren't actually creating daemons. If you have a
timed script that needs to be run you can still use a SystemStarter item
at this time. Don't get too comfortable with that though as it has been
deprecated in 10.4

plist Time


When launchd scans a folder, or a job is submitted with launchctl, it
depends on a properly formatted plist file that describes the job to be
run. This is, for most sysadmins, is the most difficult area of launchd
to grasp. SystemStarter used simple shell scripts to launch daemons.
launchd requires the admin to be able to properly format an XML file and
while not exceedingly difficult, it can be confusing.


The first stop you should make is the man page for launchd.plist. Give
it a good read and don't worry if it makes your eyes bug out. It does
that to everyone at first. At the very end it gives an example of a
plist for your perusal. This example is very simple but it breaks down
some of the keys you will become familiar with. At a minimum there are
two required launchd.plist keys. "Label" is the string that defines how
launchd will refer to the job, and "ProgramArguments" breaks down how to
execute your program. That's it, but it's not enough to generate a
useful job. Really we want to toss a few more keys in there to add to
the flexibility of the job and to make it more efficient.


Some other useful keys:
"UserName" allows you to run the job as a user other than the one who submitted it to launchd.

"inetdCompatibility" indicates that the daemon expects to be run as if it was launched by inetd.

"Program" allows you to name the path to your executable. By using
this key you can save the ProgramArguments key for, well, flags and
arguments.
"OnDemand" is a boolean that will allow you to define if a job runs continuously or not.

"RootDirectory" will let  you chroot the job into another directory.

"ServiceIPC" is where you specify if the daemon can speak IPC to launchd.

"WatchPaths" allows you to have launchd start a job based on a path modification.

"QueueDirectories" almost the same as a WatchPath, a queue will only watch an empty directory for new files.

"StartInterval" Used to schedule a job that runs on repeating schedule.

"StartCalendarInterval" You can use this to schedule jobs. The syntax is similar to cron.

"HardResourceLimits" Will allow you to restrict the resources consumed by any job.

"LowPriorityIO" Tells the kernel that this task is of a low priority when doing file system I/O.

"Sockets" This array can be used to specify what socket the daemon
will listen on for launch on demand. Check out the cool Bonjour key that
can be used to register the socket with the mDNSResponder.


Phew! That's not all of them, but I think that these represent a useful
selection. Read the man page for all of them and the options to be used.
Now that we know about some keys, lets do something neat with them.


In our example were are going to create a very simple user agent
application that moves files and folders from one directory to another.
It's a very basic sort of idea, but it's also a good building block for
more complex file processing.


First we need to come up with the script that our launchd job will execute. For our purposes something simple like:

#!/bin/bash
mv /Users/josh/in/* /Users/josh/out/
exit 0


will work. It simply takes anything in the "in" directory and moves it
to the "out" one. Stick it wherever you like. Since this is a user
specific agent I just dropped it in my home folder and named it "mover".


Now we need to create our launchd job plist. First create
~/Library/LaunchAgents, then go find a launchd plist file in
/System/Library/LaunchDeamons that seems pretty similar to what you want
to do. For this example I stole the cron plist and copied it to my new
LaunchAgents folder. Fire up your editor of choice and start making
changes.


In the end we should have something that looks a bit like this:


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.afp548.mover</string>
<key>OnDemand</key>
<true/>
<key>Program</key>
<string>/Users/josh/mover</string>
<key>ProgramArguments</key>
<array>
<string>mover</string>
</array>
<key>WatchPaths</key>
<array>
<string>/Users/josh/in</string>
</array>
</dict>
</plist>


All we really needed to do was to give it a new label, modify the
executable that is called, change the cron file's RunAtLoad key into an
OnDemand one, and make the WatchPaths point to the correct place.
Because we started with an existing file we don't need to worry about
the header or figuring the structure out on our own. Why a WatchPaths
key instead of a QueueDirectories key? The QueueDirectories depends on
the directory being empty to look for added files. If you open the
directory in the Finder it will make a .DS_Store file and throw your job
into a horrible loop. The WatchPaths is a bit different in that it
watches a path for modifications and doesn't require the directory to be
empty. WatchPaths can also keep an eye on a file. Take a look at the
default cron plist for a nice example of this.


Note that in this example I also added a ProgramArguments key. I did
this as it makes the job more flexible for future changes. Something to
keep in mind with this key is that each flag or argument to be passed to
the program must be in its own string of the array. The
org.postfix.master.plist file is a good example of this. You can't
simply say:

<key>Program</key>
<string>/usr/libexec/postfix/master -e 60</string>


but instead must use:

<key>Program</key>
<string>/usr/libexec/postfix/master</string>
<key>ProgramArguments</key>
<array>
<string>master</string>
<string>-e</string>
<string>60</string>
</array>


which can be a bit confusing at first, but is nothing too hard once you
do it the first time. Really you should take a look at the default
plists that Apple has in /System/Library/LaunchDaemons. They are full of
good examples of just about everything that you might want to do. For
example, the com.apple.periodic-daily.plist job controls running the
daily periodic job. This used to be something that was controlled by
cron, so it's a good way to see how to schedule something with launchd.
Take a look:

<key>StartCalendarInterval</key>
<dict>
<key>Hour</key>
<integer>3</integer>
<key>Minute</key>
<integer>15</integer>
</dict>


Be aware that launchd is nowhere near as flexible as Vixie cron. Luckily cron is still on the system for us to use if we wish.


If all of this seems a bit scary still, there is a nice shareware GUI
out there that can handle the file generation for you. Take a look at
the Launchd Editor
from codepoetry to see what it can do.


Now that we have our executable and plist setup how do we load our new
job into launchd? We could logout and back in, but that's a bit
intrusive to just fire up a new service. This is where launchd's buddy,
launchctl comes into play.

Enter launchctl



One of the big problems before launchd was that there was no easy or
consistent way to control system services. SystemStarter was almost
cool, but it never seemed to work properly in most cases for controlling
services after startup. The problem with the other facilities for
service control was that they are strewn across the OS with no central
way to manage them. launchctl is Apple's way of fixing this.


On its own, launchctl can take commands from the command line, from
standard in, or operate in interactive mode. If you come up with a set
of commands that you want to make permanent you can store them in
~/.launchd.conf or /etc/launchd.conf. By using these files you can setup
the environment that launchd operates in at either a user or global
level. You can use sudo launchctl to make easy changes on a global
scale, something that you couldn't do on 10.3 and earlier.


Let's start out with something simple though, loading our job. The syntax is basic, in our case:

launchctl load ~/Library/LaunchAgents/com.afp548.mover.plist



will load our job. We can confirm it's loaded with:

launchctl list



which will show us the launchd jobs that our user has submitted. If you
want to see the system job list simply run the same command with sudo.


Now that our job is loaded, try dropping something in your "in" folder.
If all is well it will vanish and then appear in your "out" folder in a
manner similar to this spiffy movie
(H264 codec required). For what it is worth, the folder in the movie
has a bunch of loose files in it and it's running on a 400 MHz G4. As
you see the response time of launchd is great. Again this is a very
simple example, but it's useful and is a great launching pad for bigger
tasks. For example launchd could watch a drop folder for graphic files
that it processes with sips and then ftp transfers to a web server. The
options are practically endless.


To remove a job we use the inverse of the load command...

launchctl unload -w ~/Library/LaunchAgents/com.afp548.mover.plist



Notice that I added the optional "-w" flag this time. This flag adds or
removes the "disabled" key from the plist at the appropriate time. If
you don't use this when unloading a job it will automatically load the
next time you login or reboot, depending on where your job file is
located. You use the same "-w" flag when loading the job to remove the
disabled key automatically.


Just like lookupd, you can also use launchctl in interactive mode.
Simply run it without any arguments and you will be delivered to the
launchd prompt. Type "help" to see the list of commands that you can
issue. They are the same as if you were calling them directly, but now
you don't need to keep typing launchctl over and over again. A standard
ctrl-d will escape the launchd shell.


If you want to view the resource usage of any particular job, or launchd
as a whole, you can use the "getrusage" command. When using this
command you need to specify if you are interested in launchd itself or
its children. Again, you use sudo to see the global resource usage. So
if I wanted to see the resource usage of all of launchd's kids:

sudo launchctl getrusage children



is all I need. If you change our demo job plist to use a
QueueDirectories key rather than the WatchPaths one there is a very high
probability that a race condition will appear. This can be really bad
as it can run wild and suck up all the CPU time on your system. Which
leads us right into our next launchctl subcommand.


Using the "limit" command we can view and set limits on the launchd
environment. With no resource specified this displays three columns
representing the resources, their soft limits, and finally, their hard
limits. Again you can use sudo to define or view these for the system as
a whole. The limits you set with launchctl are the same limits you can
set in a job plist, but they apply to the entire launchd environment
rather than a single job. There are details in the launchd.plist or
setrlimit man pages, but I'll list them briefly here:

"cpu" The maximum CPU time, in seconds, to be used by a process.

"filesize" The file size limit, in bytes, that a process may create.

"data" The data segment size limit, in bytes, for the process.

"stack" The stack segment size limit, in bytes, for the process.

"core" The core file size limit, in bytes, for the process.

"rss" This sets the resident set size that controls how much
physical memory the process may have. The OS will prefer to reclaim
memory from applications that are over their soft limit when things get
tight.
"memlock" The maximum size an app may lock into memory.

"maxproc" The maximum number of processes for a particular user ID.

"maxfiles" Obviously, the maximum number of open files for a process.




To set these limits simply use the limit command but add a resource type
and name a limit. If you only enter one number it will be used for both
the soft and hard limits. If you want them to be different you need to
enter both the soft and hard limits in order.

launchctl limit maxfiles 256 512



Will set the maxfiles limit to 512, while retaining the default soft limit of 256 files.


There are other similar launchctl commands that we can use to set
logging levels, change the stdout and stderr of launchd, set environment
variables, or change the umask of launchd. For example if I wanted to
redirect the stdout of just my launchd jobs to a file I could say:

launchctl stdout /Users/josh/Library/Logs/launchd.out



It is important to remember that these commands are job persistant, but
not launchd persistent. Meaning that the general launchd settings will
not survive a reboot. This is easy to fix by dropping a conf file in the
appropriate place. For global settings you should use /etc/launchd.conf
and for user settings you should use ~/.launchd.conf. Just enter your
launchctl commands, without the launchctl part, in the file one line at a
time. If I wanted to combine my two examples above into a permanent
setting I would put:


# This is my launchd.conf file.
limit maxfiles 256 512<br>
stdout /Users/josh/Library/Logs/launchd.out




in the appropriate launchd.conf file.

Concerns



At this point you are probably thinking that launchd is pretty cool and you would be right, but I do have a few concerns.


First of all launchd replaced init and xinetd with one process. This is a
bit scary as we now basically have init listening in a bunch of
different ways for something to tell it to start a job. The security
implications of this aren't really known yet with launchd being as young
as it is.


Secondly, and in the same vein, launchd is process 1 and it has the
potential to take down the whole system. I've already seen unconfirmed
reports of a ssh scan on a network causing launchd to freak out and make
systems inaccessible. Having at least some sort of resource limit set
on jobs might help here.


These risks and unknowns could have been minimized by gradually phasing
launchd in, but Apple has chosen to drop it on us like a bomb. I know of
people who are migrating services back to xinetd when they do installs.
It's not because launchd is flawed, but because it is a bit of an
unknown for the time being.


That said I'm really excited by the potential of launchd. We are now one GUI away from a Windows-style "Services" utility.

Wrapping up



Wow, that was a lot to take in at once eh? launchd is here, and it is
the future of service management on Mac OS X. Apple has opened the
project in the hopes that other *NIX vendors will take a look at it and
incorporate it into their own OSes. With launchd being the work of
Jordan Hubbard it probably carries a bit more clout than other Apple OSS
projects in the community. All in all it is the most aggressive attempt
by anyone to reign in the herd of cats that booting and maintaining a
UNIX system has become, and for that we should be standing and cheering.


As always, have fun and read the man pages.

man launchd

man launchd.conf


man launchd.plist

man launchctl

man launchd_debugd



That last one is a nice little nugget, but it is a bit of a trick as the
plist it references isn't installed by default. First you need to get
the xml file from the darwin source here
(Apple ID login required). Change its name from
com.apple.launchdebugd.xml to com.apple.launchd_helperd.plist, copy it
to /System/Library/LaunchDaemons, load it, and then point your web
browser at port 12345 on the Mac in question to get a cool output of all
loaded jobs and their settings.


(If you don't have an Apple ID, or don't want to agree to the Apple Open
Source license, I've put a copy of the finished launchd_helperd.plist
in our file database.
)


(Ed. note:
For the ease of sharing, this document is released with the  GFDL
license as well as our regular  Creative Commons
license.)





from:http://www.afp548.com/article.php?story=20050620071558293