Week 3.b CS5600 1/25 2023 https://naizhengtan.github.io/23spring/ 1. Survey 2. Process birth 3. Shell crash course 4. Shell internals I 5. File descriptors 6. Shell internals II ---------- 1. Survey [show results] 2. Process birth How does a process come into being? --answer: a system call! --in Unix, it is fork() --fork() creates an almost exact copy, except that the return value is different --QUESTION: what's wrong with an exact copy? [think as a process] --parent/child example if (fork() == 0) { // child } else { // parent } --history: 1962, Melvin Conway, "as an intellectual exercise" --original: fork-join [draw fork-join on board] --now, we have two syscalls, fork() and wait() --Q: Can we create a process by "create_process" (a syscall)? --that's what windows do --why fork() works "better"? # why fork() is less attractive today? --wait(int *wstatus) --will block until one child terminates --"wstatus" collects the status (e.g., exit code) of the child --what if a child process finishes but the parent process never call wait()? [answer: child process becomes zombie process] --what if a parent process finishes before a child process? [answer: child process becomes orphan process] --QUESTION: zombie and orphan, which do you think is more problematic? [answer: zombie is more annoying because they will consume system resources, the process table.] 3. Shell crash course - how fork/wait works in reality? -- think of how you run your Lab1 program - a program that creates processes - the human's interface to the computer - GUIs (graphical user interfaces) are another kind of shell. - mechanically introduce (to some of you, review) the basics of shell -- next section will discuss their motivations, how they are implemented, and why they are powerful - shell reads user inputs (cmds and arguments) and run the cmds for example, "$ ls" and "$ ls -a" -- "ls" is a program, like your hellowrold -- "-a" is an argument; you will see argument handling in Lab1 - output redirection -- "$ ls" prints to screen; what if I want the output to a file? -- Question: how people will do this in Windows? -- "$ ls > files.txt" - backgrounding -- there are long-running jobs, for example, web server -- run them by backgrounding $ web-server & $ - pipe -- feed one program's output to another's input -- "$ cat students.txt | shuf -n 1" -- equivalent to "$ cat students.txt > /tmp/tmpfile $ shuf -n 1 /tmp/tmpfile $ rm /tmp/tmpfile" (of course, technically, we don't need "cat" and "/tmp/tmpfile") - Shell builtin cmds vs. program -- "echo/pwd/which" vs. "ls" -- use "which" to tell program: "$ which ls" => "/bin/ls" built-in: "$ which which" => "which: shell built-in command" -- why builtin cmds? -- has to be builtin (impossible to implement otherwise): "cd" -- for efficiency: "echo" ----- Git and Lab grading explained - if you know why, usually things are not going to be too bad [draw the workflow of students student -> file -> git tracked file -> committed file -> pushed file student --commit id--> Canvas slack hours ] [draw grading workflow Canvas --commit id--> TA computer TA computer --commit id--> GitHub --code--> TA computer TA computer compile, run, grade Loop 60+ times ] Q: I have a new file. Should I commit it? modified Makefile? [Yes, otherwise, we cannot compile your code.] Q: If I accidentally commit .o file or binary, am I doomed? [No, that's okay. We will run make anyway.] Q: Can I modify skeleton code? [Yes. In fact, you can complete delete the entire project and start from scratch.] Q: Can I change the shell API? [No, you will get a zero.] [Slack hours] - anyone who don't understand slack hours? - a useful question: A student finished Lab3. Now is 60hrs late. The student only has 10 slack hrs. How many hours should the student apply to this lab? [answer: 0hr; floor is 50pt] - repeat that we will run cheating detection system. a note: - distinguish two things: what you've learned (self improvment) vs. lab/final grades [they not necessarily align. In fact, in many cases, they conflict.] - this is an optimization problem max SOMETHING s.t. time --know your optimization object --know your constraints ----- 4. Shell internals, part I a. How does the shell start programs? --example: $ ls [see panel 1 on handout; go line-by-line] --calls fork(), which creates a copy of the shell. now there are two copies of the shell running --then calls exec(), which loads the new program's instructions into memory and begins executing them. --(exec invokes the loader) while (1) { write(1, "$ ", 2); readcommand(command, args); // parse input if ((pid = fork()) == 0) // child? execve(command, args, 0); else if (pid > 0) // parent? wait(0); //wait for child else perror("failed to fork"); } [why 0 in wait(0)? 0 means NULL] --waits for the end of a process --with wait() or waitpid() system calls --QUESTION: why is fork different from exec? why not combine them? [in fact, we have "posix_spawn()" to somewhat simulate fork/exec] * We will come back to this. this => "the power of the fork/exec separation" b. Redirection and pipe, motivation Q: in Windows, how can you concate two txt files? 10 files? 1000 files? What does this do? $ cat file1 file2 > new_file or say we wanted to extract all of your GitHub ids...how would you do that without pipelines? fetch repos from html page https://github.com/NEU-CS5600-23spring (call it URL) then $ curl $URL | grep -o lab1-[a-zA-Z0-9\-]* | uniq > repo.txt How are these things implemented? Remember, the programmer of cat is long gone, and their output is winding up somewhere that the original program never specified. 5. File descriptors --"int fd = open(const char* path, int flags)" --every process can usually expect to begin life with three file descriptors already open: 0: represents the input to the process (e.g., tied to terminal) 1: represents the output 2: represents the error output these are sometimes known as stdin, stdout, stderr --NOTE: Unix hides for processes the difference between a device and a file. this is a very powerful hiding (or abstraction), as we will see soon [draw kernel file descriptor table] 6. Shell internals, part II - redirection Back to $ cat abcd efgh > /tmp/foo How is that implemented? Answer: after fork() but before exec(), shell does: close(1) open("/tmp/foo", O_TRUNC | O_CREAT | O_WRONLY, 0666) which automatically assigns fd 1 to point to /tmp/foo [draw picture of fds, fd 0 /dev/tty, fd 1 now /tmp/foo] --now, when "cat" runs, it still has in its code: write(1,...), but "1" now means something else. [read handout part 2] What about $ sh < script > tmp1 where script contains echo abc echo def [draw picture] - Pipelines [start from here next time]