Week 3.a
CS5600
1/31 2022
https://naizhengtan.github.io/22spring/

1. Last time
2. Process birth
3. Shell crash course
4. Shell internals I
5. File descriptors
6. Shell internals II

---------------------------------------------------------------------------

Admin:

- Lab0 due today; Lab1 will be released tomorrow

- HW1 due Wed; no late submission

- review session this week
  -- gdb and vim
  -- Thursday, 13:00-14:00

- textbook & lecture & labs

- online lectures
  -- slightly faster than in-person ones; no break
  -- ask questions: either on chat (TA will relay) or unmute and ask

---------------------------------------------------------------------------


1. last time

  - a question about handout week2.a
    -- why compiler allocates more stack space in f's frame?
    -- answer: 
       We use "-O0" which means "no optimization".
       Also, it (how much memory to allocate) is compiler’s leeway.
       As far as the assembly code is functional correctly,
       the compiler follows its spec.

    -- "gcc -O3"

    f:
    subq    $24, %rsp        // push frame: allocate stack frame 24B

                             // [Security]
    movq    %fs:40, %rax     // canary value (%fs:40) for detecting stack smashing attacks
    movq    %rax, 8(%rsp)    // put canary at 8(%rsp)

    xorl    %eax, %eax       // %eax=0 [%eax is the low 32bit of %rax]
    movq    %rsp, %rdx       // %rdx = %rsp
    movq    (%rdi), %rax     // %rax = *ptr (%rdi contains the first arg to function f, which is "ptr")
    movq    %rdx, q(%rip)    // "q" is the global variable "q"; set the %rdx to global variable "q"

                             // [Security]
    movq    8(%rsp), %rcx    // move the canary at stack, namely 8(%rsp) to %rcx
    xorq    %fs:40, %rcx     // check if the value has been changed
    jne    .L9               // if so, alert!!!

    leaq    1(%rax,%rax), %rax  //  %rax = %rax + %rax + 1 (!!! this is function g!)
    addq    $24, %rsp           // pop frame
    ret                         // return to main


    [For those who are interested in assembly:
      https://web.stanford.edu/class/archive/cs/cs107/cs107.1166/guide_x86-64.html]

 Process review

  [draw CPU+memory, then process' view, then registers, then frames, then
  kernel, then syscalls/exceptions/interrupts]

  - process, an abstraction of machine
    -- registers & memory

  - stack frames
    -- assembly: push/pop/call/ret
    -- registers: %rip, %rbp, %rsp
    -- function prologue, epilogue
    -- Calling conventions

  - variables in memory
    -- local variables => stack
    -- global variables => data segment
    -- malloc/free => heap
      -- why
      -- HOWTO

  - system calls
    -- interface between processes and kernel

  - three ways of process-kernel control transfer
    -- system calls, interrupts, and exceptions

  - an evaluation
    -- given a piece of code,
    -- you can simulate CPU & manage memory (in particular, the stack)
    -- to executes the program.
    -- [you're a process Ninja if you can achieve this.]


2. Process birth

    How does a process come into being?

    --answer: a system call!

    --in Unix, it is fork()

    --fork() creates an almost exact copy, except that the return value
      is different

    --QUESTION: what's wrong with an exact copy?
      [think as a process]

    --parent/child example

      if (fork() == 0) {
          // child
      } else {
          // parent
      }

    --history: 1962, Melvin Conway, "as an intellectual exercise"
      --original: fork-join
      [draw fork-join on board]

      --now, we have two syscalls, fork() and wait()

    --why not create a process by "create_process"?
      --that's what windows do
      --why fork() works "better"?
      # why fork() is less attractive today?

    --wait(int *wstatus)
      --will block until one child terminates
      --"wstatus" collects the status (e.g., exit code) of the child

    --what if a child process finishes but the parent process
      never call wait()?
      [answer: child process becomes zombie process]

    --what if a parent process finishes before a child process?
      [answer: child process becomes orphan process]

    --QUESTION: zombie and orphan, which do you think is more problematic?
      [answer: zombie is more annoying because they will consume system
      resources, the process table.]


3. Shell crash course

  - how fork/wait works in reality?
    -- think of how you run a "helloworld" program

  - a program that creates processes

  - the human's interface to the computer

  - GUIs (graphical user interfaces) are another kind of shell.

  - mechanically introduce (to some of you, review) the basics of shell
    -- next section will discuss their motivations, how they
       are implemented, and why they are powerful

  - shell reads user inputs (cmds and arguments) and run the cmds
    for example, "$ ls" and "$ ls -a"
    -- "ls" is a program, like your hellowrold
    -- "-a" is an argument; you will see argument handling in Lab1

  - output redirection
    -- "$ ls" prints to screen; what if I want the output to a file?

    -- Question: how people will do this in Windows?

    -- "$ ls > files.txt"

  - backgrounding
    -- there are long-running jobs, for example, web server
    -- run them by backgrounding
      $ web-server &
      $

  - pipe
    -- feed one program's output to another's input
    -- "$ cat students.txt | shuf -n 1"
    -- equivalent to 
       "$ cat students.txt > /tmp/tmpfile
        $ shuf -n 1 /tmp/tmpfile
        $ rm /tmp/tmpfile"
       (of course, technically, we don't need "cat" and "/tmp/tmpfile")

  - Shell builtin cmds vs. program
    -- "echo/pwd/which" vs. "ls"
    -- use "which" to tell
       program:  "$ which ls"    => "/bin/ls"
       built-in: "$ which which" => "which: shell built-in command"
    -- why builtin cmds?
      -- has to be builtin (impossible to implement otherwise): "cd"
      -- for efficiency: "echo"


4. Shell internals, part I

    a. How does the shell start programs?

    --example:
        $ ls

    [see panel 1 on handout; go line-by-line]

    --calls fork(), which creates a copy of the shell. now there are
    two copies of the shell running

    --then calls exec(), which loads the new program's instructions
    into memory and begins executing them.
        --(exec invokes the loader)

    while (1) {
        write(1, "$ ", 2);
        readcommand(command, args); // parse input
        if ((pid = fork()) == 0) // child?
            execve(command, args, 0);
        else if (pid > 0) // parent?
            wait(0); //wait for child
        else
            perror("failed to fork");
    }

    [why 0 in wait(0)? will be homework]

    --waits for the end of a process
        --with wait() or waitpid() system calls

    --QUESTION: why is fork different from exec?
      why not combine them?

       [in fact, we have "posix_spawn()" to somewhat simulate fork/exec]

      * We will come back to this.
      this => "the power of the fork/exec separation"

    b. Redirection and pipe, motivation

        What does this do?

        $ cat abcd efgh > foo

        or say we wanted to extract all of your GitHub ids...how would
        you do that without pipelines?

        download html from https://github.com/NEU-CS5600-22spring (call it URL)
        then
        $ curl $URL | grep -o lab0-[a-zA-Z0-9\-]* | uniq | shuf -n 1

        How are these things implemented? Remember, the programmer of
        cat is long gone, and their output is winding
        up somewhere that the original program never specified.

5. File descriptors

    --"int fd = open(const char* path, int flags)"

    --every process can usually expect to begin life with three file
    descriptors already open:
    0: represents the input to the process (e.g., tied to
        terminal)
    1: represents the output
    2: represents the error output

    these are sometimes known as stdin, stdout, stderr

    --NOTE: Unix hides for processes the difference between a device and
    a file. this is a very powerful hiding (or abstraction), as we will
    see soon

6. Shell internals, part II

    - redirection

    Back to 
         $ cat abcd efgh > /tmp/foo

  How is that implemented?

    Answer: after fork() but before exec(), shell does:

            close(1)
            open("/tmp/foo", O_TRUNC | O_CREAT | O_WRONLY, 0666)

        which automatically assigns fd 1 to point to /tmp/foo

        [draw picture of fds, fd 0 /dev/tty, fd 1 now /tmp/foo]

        --now, when "cat" runs, it still has in its code: write(1,...),
        but "1" now means something else.

    [read handout part 2]

    - The power of the fork/exec separation

    [an innovation from the original Unix. possibly lucky design
    choice at the time. but turns out to work really well.  allows
    the child to manipulate environment and file descriptors
    *before* exec, so that the *new* program may in fact encounter a
    different environment]

      --recall how we handle redirection

      --To generalize redirections and pipelines, there are lots of
      things the parent shell might want to manipulate in the child
      process: file descriptors, environment, resource limits.

      --yet fork() requires no arguments!

      --Contrast with CreateProcess on Windows:

       BOOL CreateProcess(
       name,
       commandline,
       security_attr,
       thr_security_attr,
       inheritance?,
       other flags,
       new_env,
       curr_dir_name,
       .....)

       [http://msdn.microsoft.com/en-us/library/ms682425(v=VS.85).aspx]

       there's also CreateProcessAsUser, CreateProcessWithLogonW,
       CreateProcessWithTokenW, ...

    * The issue is that any conceivable manipulation of the
    environment of the new process has to be passed through 
    arguments, instead of via arbitrary code.

    in other words:

      because whoever calls CreateProcess() (or its variant) needs
      to perfectly configure the process before it starts running.

      with fork(), whoever calls fork() **is still running** so
      can arrange to do whatever it wants, without having to work
      through a rigid interface like the above. allows arbitrary
      "setup" of the process before exec().

    - fork() is less attractive today

      [briefly talked about reasons; will discuss in the next lecture]

      "A fork() in the road"
      https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf


Aside:

     - Fork bomb at the bash command prompt:
            $ :(){ : | : & }; :