Jekyll2023-09-25T19:01:35+00:00http://hongyu.nl/feed.xmlHongyu HèBlogHongyu HèQueuing Theory: Understanding Waiting Lines2023-02-05T00:00:00+00:002023-02-05T00:00:00+00:00http://hongyu.nl/queuing<p>Queuing theory is a mathematical concept that deals with the study of queues or waiting lines. Queuing theory has found significant applications in various fields, including computer systems. In computer systems, queuing theory helps in analyzing the performance, predicting the system’s behavior under different load conditions, and optimizing system parameters for better performance. In this blog, we will discuss queuing theory in computer systems and explain some of the essential formulas used in queuing theory.</p> <h2 id="basics-of-queuing-theory"><strong>Basics of Queuing Theory</strong></h2> <p>Queuing theory is a branch of applied mathematics that deals with the study of queues or waiting lines. Queues are a common phenomenon in everyday life, and we encounter them in various situations, such as waiting for a bus, standing in a line at a grocery store, or waiting for a web page to load. In computer systems, queuing theory helps in analyzing the behavior of computer systems under different load conditions.</p> <p>A queuing system consists of three basic components:</p> <ul> <li>The population of customers who need service</li> <li>The service facility or server that provides the service</li> <li>The waiting line or queue where customers wait for service</li> </ul> <p>In a queuing system, customers arrive randomly, and they join the queue if the service facility is busy. When a customer arrives at the service facility, the service facility serves the customer, and the customer leaves the system. The queuing system’s performance can be measured using various metrics, such as the average waiting time, the average queue length, the utilization of the service facility, and the throughput.</p> <h2 id="kendalls-notation"><strong>Kendall’s Notation</strong></h2> <p>Kendall’s notation is a standard notation used to describe queuing systems. Kendall’s notation uses four letters to describe a queuing system. The first letter represents the arrival process, the second letter represents the service process, the third letter represents the number of servers, and the fourth letter represents the queue discipline.</p> <p>The following table summarizes the meaning of each letter in Kendall’s notation:</p> <table> <thead> <tr> <th>Letter</th> <th>Meaning</th> </tr> </thead> <tbody> <tr> <td>$A$</td> <td>Arrival process</td> </tr> <tr> <td>$M$</td> <td>Markovian (exponential) service process</td> </tr> <tr> <td>$D$</td> <td>Deterministic service process</td> </tr> <tr> <td>$G$</td> <td>General service process</td> </tr> <tr> <td>$M/D/C$</td> <td>Number of servers ($C$)</td> </tr> <tr> <td>$F$</td> <td>Queue discipline</td> </tr> </tbody> </table> <h2 id="formulas-in-queuing-theory"><strong>Formulas in Queuing Theory</strong></h2> <p>The following are some essential formulas used in queuing theory:</p> <h3 id="littles-law"><strong>Little’s Law</strong></h3> <p>Little’s law states that the average number of customers in a queuing system is equal to the product of the average arrival rate and the average time a customer spends in the system. Mathematically, Little’s law can be expressed as follows:</p> $L = \lambda W$ <p>where $L$ is the average number of customers in the system, $λ$ is the arrival rate, and $W$ is the average time a customer spends in the system.</p> <h3 id="erlangs-formula"><strong>Erlang’s Formula</strong></h3> <p>Erlang’s formula is used to calculate the probability of a customer having to wait in the queue before receiving service. Erlang’s formula assumes that the arrival process is Poisson, the service process is Markovian, and there is only one server. Mathematically, Erlang’s formula can be expressed as follows:</p> $P_n = \frac{\frac{(\lambda/\mu)^n}{n!}}{\sum_{i=0}^{C-1} \frac{(\lambda/\mu)^i}{i!} + \frac{(\lambda/\mu)^C}{C!(1-\rho)}}$ <p>where $P_n$ is the probability of $n$ customers in the system, $λ$ is the arrival rate, $μ$ is the service rate, $C$ is the number of servers, and $ρ$ is the utilization of the service facility.</p> <h3 id="kendalls-formulas"><strong>Kendall’s Formulas</strong></h3> <p>Kendall’s formulas are used to calculate the performance measures of queuing systems. The following are the most commonly used Kendall’s formulas:</p> <ul> <li>Average queue length:</li> </ul> $L_q = \frac{\rho^2}{1-\rho} \cdot \frac{1}{1-C\rho^{-C}(1-\rho)^{-1}}$ <p>where $L_q$ is the average queue length, $ρ$ is the utilization of the service facility, and $C$ is the number of servers.</p> <ul> <li>Average waiting time:</li> </ul> $W_q = \frac{L_q}{\lambda}$ <p>where $W_q$ is the average waiting time and $λ$ is the arrival rate.</p> <ul> <li>Utilization:</li> </ul> $\rho = \frac{\lambda}{C\mu}$ <p>where $ρ$ is the utilization of the service facility, $λ$ is the arrival rate, $μ$ is the service rate, and $C$ is the number of servers.</p> <ul> <li>Throughput:</li> </ul> $X = \lambda(1-P_n)$ <p>where $X$ is the throughput, $λ$ is the arrival rate, and $P_n$ is the probability of $n$ customers in the system.</p> <h2 id="applications-of-queuing-theory">Applications of Queuing Theory</h2> <p>Queuing theory has many practical applications in various fields. In telecommunications, queuing theory is used to optimize the performance of call centers and customer service systems. By using queuing theory, call centers can determine the optimal number of agents needed to handle customer demand and minimize waiting times. In the healthcare sector, queuing theory is used to manage patient flows and optimize hospital resources. By understanding the behavior of queues, hospitals can reduce waiting times, improve patient satisfaction, and increase the efficiency of their operations.</p> <p>In manufacturing, queuing theory is used to optimize production lines by minimizing queue lengths and waiting times. By analyzing the arrival and service rates of a production line, queuing theory can help manufacturers determine the optimal number of servers (e.g., machines) needed to meet customer demand and reduce waiting times. Queuing theory is also used in inventory management to determine the optimal inventory level that minimizes costs while meeting customer demand.</p> <p>Queuing theory is also useful in traffic engineering, where it is used to optimize the performance of traffic systems. By understanding the behavior of queues, traffic engineers can design traffic systems that minimize congestion and waiting times. For example, queuing theory can help traffic engineers determine the optimal number of traffic lanes needed to handle traffic flows during peak hours and minimize waiting times.</p> <p>Queuing theory has been used in the entertainment industry to predict demand for rides and attractions at theme parks. By analyzing the arrival rate and service rate, queuing theory can help theme parks determine the optimal number of employees needed to reduce queue lengths and waiting times. Queuing theory has also been applied in the aviation industry to optimize the allocation of gates and reduce waiting times at airports.</p> <h2 id="limitations-of-queuing-theory">Limitations of Queuing Theory</h2> <p>While queuing theory is a powerful tool for understanding waiting lines, it has some limitations. Queuing theory assumes that customers arrive randomly and independently of each other, which may not always be the case in real-world scenarios. For instance, customers may arrive in groups or bunches, and their arrival may be dependent on external factors like weather, time of day, or season. Additionally, queuing theory assumes that the service process is independent of the arrival process, which may not always hold true. In some cases, the arrival of customers may be dependent on the state of the queue or the number of customers present. Finally, queuing theory assumes that the queue discipline is fixed, which may not be the case in real-world scenarios where priorities may change. For example, in a hospital, the priority of patients may change based on their medical condition.</p> <h2 id="outlook">Outlook</h2> <p>Queuing theory is a dynamic field that continues to evolve as new applications and challenges arise. With the advent of big data and machine learning, queuing theory is now being used to optimize complex systems that were previously difficult to model. For example, queuing theory is being used to optimize cloud computing systems, where the arrival rate and service rate can vary significantly depending on the workload. Queuing theory is also being used to optimize supply chain management, where the arrival rate and service rate can vary depending on the demand for goods and services.</p> <p>Another area of interest for future research in queuing theory is the study of the impact of social distancing measures on queues. The COVID-19 pandemic has brought about significant changes in the way we wait in lines. Queuing theory can be used to study the effectiveness of social distancing measures in reducing queue lengths and waiting times.</p> <p>Furthermore, queuing theory is being applied to study the impact of customer behavior on queuing systems. It is being used to analyze the effect of customer impatience, customer balk, and jockeying behavior on queueing systems. By understanding the behavior of customers, queuing theory can help organizations design better queue management systems that cater to the needs and preferences of their customers.</p> <h2 id="summary">Summary</h2> <p>Queuing theory is a methematical concept for understanding waiting lines and improving their performance. By applying queuing theory to real-world problems, we can optimize the performance of queues in various applications, from customer service systems to traffic systems. Queuing theory provides a quantitative framework for evaluating the performance of queues and improving their efficiency, enhancing the experience of customers and users. While queuing theory has its limitations, it remains an essential tool for analyzing and optimizing waiting lines in a variety of fields. As the world becomes more complex, queuing theory will continue to play a critical role in optimizing systems and improving the efficiency of operations.</p> <p>In addition to the above, queuing theory is also being increasingly applied in the field of e-commerce, where it is used to optimize online shopping experiences. With the exponential growth of online shopping, queuing theory is being used to reduce the waiting times for customers during peak periods, such as Black Friday and Cyber Monday. By analyzing the behavior of online shoppers, queuing theory can help retailers to design better queuing systems that can accommodate the surge in demand during such peak periods.</p> <p>Another area in which queuing theory is being applied is in the field of public transportation. Queuing theory is used to optimize public transportation systems by minimizing waiting times at bus stops and train stations. By analyzing the arrival and service rates of public transportation systems, queuing theory can help transit agencies to determine the optimal number of buses or trains needed to meet demand and reduce waiting times for passengers.</p> <p>Queuing theory is also being used to improve the performance of online streaming services. By analyzing the arrival and service rates of streaming services, queuing theory can help streaming providers to determine the optimal number of servers needed to handle the demand for streaming content and reduce buffering times for users.</p> <p>In conclusion, queuing theory is a powerful tool that has numerous applications in various fields. As the world becomes more complex, the need for efficient queue management systems becomes even more paramount. By understanding the behavior of queues, organizations can optimize their operations, reduce waiting times, and enhance the experience of their customers and users. Queuing theory will continue to play a critical role in optimizing systems and improving the efficiency of operations in various fields.</p> <p>Moreover, the application of queuing theory has expanded to the field of public health. During the COVID-19 pandemic, queuing theory has been used to model the spread of the virus and to predict the impact of social distancing measures on the spread of the virus. Queuing theory has been used to model the behavior of the virus in different populations and to predict the effectiveness of various interventions, such as lockdowns and vaccination programs. By understanding the behavior of the virus and the effectiveness of interventions, queuing theory can help public health officials to design better policies and strategies to combat the spread of the virus.</p> <p>In addition to the above-mentioned applications, queuing theory is also being used in the field of finance. Queuing theory is used to optimize the performance of financial markets by minimizing waiting times and reducing transaction costs. By analyzing the arrival and service rates of financial markets, queuing theory can help investors to determine the optimal timing and size of their trades and to minimize their losses due to transaction costs.</p> <p>Another area of interest for future research in queuing theory is the study of the impact of emerging technologies on queue management systems. With the rapid pace of technological innovation, queuing theory can help organizations to design better queue management systems that can accommodate new technologies such as artificial intelligence, robotics, and the internet of things. By understanding the behavior of queues in the context of emerging technologies, queuing theory can help organizations to optimize their operations and improve the efficiency of their systems.</p> <h2 id="reference">Reference</h2> <ul> <li>Gross, D. and Harris, C. M. (1998) Fundamentals of Queuing Theory. John Wiley &amp; Sons, Inc.</li> <li>Hillier, F. S. and Lieberman, G. J. (2014) Introduction to Operations Research, 10th edition. McGraw-Hill Education.</li> <li>Kleinrock, L. (1975) Queueing Systems, Volume 1: Theory. John Wiley &amp; Sons, Inc.</li> <li>Takács, L. (1962) Introduction to the Theory of Queues. Oxford University Press.</li> <li>Whitaker, R. (1998) The Queuing Network Analyzer: A User’s Guide. McGraw-Hill Education.</li> </ul>Hongyu Hèsystems theoryA Glimpse of Survival Analysis2023-01-17T00:00:00+00:002023-01-17T00:00:00+00:00http://hongyu.nl/survival<p>Survival analysis is a statistical method used to analyze the time until an event of interest occurs. It is used in various fields, including medical research, engineering, and social sciences. In this blog, we will explore the basics of survival analysis, including the definition of survival function, hazard function, and Kaplan-Meier estimator.</p> <h3 id="definition-of-survival-function"><strong>Definition of Survival Function</strong></h3> <p>The survival function is a fundamental concept in survival analysis. It describes the probability that an individual survives beyond a certain time. Mathematically, the survival function is defined as:</p> $S(t) = P(T &gt; t)$ <p>where $T$ is the random variable representing the time until the event of interest occurs, and $t$ is a specific time point. The survival function can also be interpreted as the proportion of individuals who survive beyond time $t$.</p> <h3 id="hazard-function"><strong>Hazard Function</strong></h3> <p>The hazard function is another important concept in survival analysis. It describes the instantaneous rate of occurrence of the event of interest at time $t$, given that the individual has survived up to time $t$. The hazard function is defined as:</p> $h(t) = \lim_{\Delta t \rightarrow 0} \frac{P(t \leq T &lt; t + \Delta t | T \geq t)}{\Delta t}$ <p>The hazard function can also be interpreted as the probability of the event of interest occurring in the next infinitesimal time interval, given that the individual has survived up to time $t$.</p> <h3 id="kaplan-meier-estimator"><strong>Kaplan-Meier Estimator</strong></h3> <p>The Kaplan-Meier estimator is a non-parametric method used to estimate the survival function. It is particularly useful when the distribution of survival times is unknown or non-normal. The estimator is based on the observed survival times of a sample of individuals. Let $t_1, t_2, …, t_n$ be the observed survival times, and let $d_1, d_2, …, d_n$ be the corresponding number of events (i.e., deaths) at each time point. The Kaplan-Meier estimator is defined as:</p> $\hat{S}(t) = \prod_{i:t_i \leq t} \left( 1 - \frac{d_i}{n_i} \right)$ <p>where $n_i$ is the number of individuals at risk at time $t_i$. The estimator can be interpreted as the product of the probabilities of survival up to each time point. The denominator in the product is the number of individuals at risk at each time point, which is equal to the total sample size minus the number of events up to that time point.</p> <h3 id="log-rank-test"><strong>Log-Rank Test</strong></h3> <p>The log-rank test is a statistical test used to compare the survival distributions of two or more groups. The null hypothesis is that the survival distributions are the same across all groups. The test is based on the difference between the observed and expected number of events in each group, assuming that the null hypothesis is true. The test statistic is given by:</p> $Z = \frac{(O_1 - E_1)^2}{V_1} + \frac{(O_2 - E_2)^2}{V_2} + ... + \frac{(O_k - E_k)^2}{V_k}$ <p>where $O_i$ is the observed number of events in group $i$, $E_i$ is the expected number of events in group $i$ under the null hypothesis, and $V_i$ is the variance of the number of events in group $i$ under the null hypothesis. The test statistic follows a chi-squared distribution with $k-1$ degrees of freedom, where $k$ is the number of groups being compared.</p> <h3 id="cox-proportional-hazards-model"><strong>Cox Proportional Hazards Model</strong></h3> <p>The Cox proportional hazards model is a widely used semi-parametric method in survival analysis. It is used to model the relationship between covariates (i.e., explanatory variables) and the hazard function, while assuming that the hazard ratios are constant over time. The hazard ratio represents the relative risk of the event of interest for individuals with a certain value of the covariate, compared to individuals with a reference value of the covariate. The Cox model assumes that the hazard function is given by:</p> $h(t, \boldsymbol{X}) = h_0(t) \exp(\beta_1 X_1 + \beta_2 X_2 + ... + \beta_p X_p)$ <p>where $h_0(t)$ is the baseline hazard function (i.e., the hazard function for individuals with a reference value of all covariates), $\boldsymbol{X}$ is a vector of $p$ covariates, and $\beta_1, \beta_2, …, \beta_p$ are the corresponding regression coefficients. The Cox model does not require any assumptions about the shape of the baseline hazard function, making it more flexible than parametric models.</p> <p>The likelihood function for the Cox model can be written as:</p> $L(\beta_1, \beta_2, ..., \beta_p) = \prod_{i=1}^n \left( \frac{\exp(\beta_1 X_{i1} + \beta_2 X_{i2} + ... + \beta_p X_{ip})}{\sum_{j \in R_i} \exp(\beta_1 X_{j1} + \beta_2 X_{j2} + ... + \beta_p X_{jp})} \right)^{\delta_i}$ <p>where $X_{i1}, X_{i2}, …, X_{ip}$ are the covariate values for individual $i$, $\delta_i$ is the event indicator for individual $i$ (i.e., $\delta_i=1$ if the event of interest occurs for individual $i$, and $\delta_i=0$ otherwise), and $R_i$ is the set of individuals at risk at the time of event for individual $i$. The Cox model estimates the regression coefficients that maximize the likelihood function.</p> <h3 id="summary"><strong>Summary</strong></h3> <p>Survival analysis is a powerful tool for analyzing time-to-event data. The survival function and hazard function are important concepts in survival analysis, and the Kaplan-Meier estimator is a useful non-parametric method for estimating the survival function. The log-rank test and Cox proportional hazards model are commonly used statistical methods for comparing survival distributions and modeling the relationship between covariates and the hazard function, respectively. By understanding these concepts and methods, researchers can gain valuable insights into the time-to-event outcomes in their data.</p>Hongyu Hèsystems theoryAre you Sure your Linux PID is the Process ID?2023-01-01T00:00:00+00:002023-01-01T00:00:00+00:00http://hongyu.nl/pid<h2 id="ids-for-processes-and-threads-in-linux">IDs for Processes and Threads in Linux</h2> <ol class="text-left"> <li>In Linux, all processes and threads get a unique identifier (ID) and are all listed as directories <em>of the same level</em> under the <code class="language-plaintext highlighter-rouge">/proc</code> pseudo file system.</li> <li>These processes and threads are represented as subdirectories in the form of <code class="language-plaintext highlighter-rouge">/proc/[pid]</code> <em>,</em> where the <code class="language-plaintext highlighter-rouge">pid</code> is the unique numerical ID for each.</li> <li>So, technically, the “<code class="language-plaintext highlighter-rouge">pid</code>” is not just a process ID but could be an ID for either a thread or a process.</li> <li><code class="language-plaintext highlighter-rouge">htop</code> shows both processes and threads, and doesn’t distinguish them by default. <ul> <li>Consequently, its “PID” column contains IDs for both processes and threads.</li> </ul> </li> <li>This dreadful terminology overloading has a historical reason: <ul> <li>Linux originally didn’t have the notion of threads.</li> <li>It only had separate processes, each of which has a unique <code class="language-plaintext highlighter-rouge">pid</code>, potentially sharing some resources like virtual memory and file descriptors.</li> <li> <p>In 2001, Linux 2.4 introduced “Thread groups”, which gave rise to threads within a process. From the <a href="http://man7.org/linux/man-pages/man2/clone.2.html">clone(2) man page</a>:</p> <blockquote> <p><em>Thread groups were a feature added in Linux 2.4 to support the POSIX threads notion of a set of threads that share a single PID. Internally, this shared PID is the so-called thread group identifier (TGID) for the thread group. Since Linux 2.4, calls to <code class="language-plaintext highlighter-rouge">getpid(2)</code> return the TGID of the caller.</em></p> </blockquote> <ul> <li>So, the PID of the parent process is overloaded with another meaning, the TGID :)</li> </ul> </li> <li>That’s said, the kernel still doesn’t have a separate implementation for processes and threads.</li> </ul> </li> <li>In summary, threads within a process will get the <em>same</em> PID , while each of which has a unique thread ID (TID). <ul> <li>The values returned by the <code class="language-plaintext highlighter-rouge">getpid()</code> are the same for all of them.</li> <li>But the TID from the <code class="language-plaintext highlighter-rouge">gettid()</code> are always unique.</li> </ul> </li> </ol> <h2 id="how-to-distinguish-between-a-thread-and-a-real-process">How to Distinguish between a Thread and a real “Process”?</h2> <ol class="text-left"> <li>As mentioned above, threads and process are similar in nature (kernel implementation), and both are “schedulable” entities to the kernel.</li> <li>The main difference between them is the scope of their namespaces (address space, resources, etc.).</li> <li>To distinguish between threads and processes, we need to look into the <code class="language-plaintext highlighter-rouge">/proc/[pid]/task/[tid]</code> subdirectories where <code class="language-plaintext highlighter-rouge">tid</code> is the <em>kernel</em> <em>thread ID</em>.</li> <li>Kernel threads are the ones registered in and scheduled by the kernel. <ul> <li>There are user-level threads (e.g., Java threads) invisible to the kernel and managed by the application layer.</li> <li>See <a href="https://www.ibm.com/docs/da/aix/7.2?topic=processes-kernel-threads-user-threads">here</a> for more.</li> </ul> </li> <li>The threads within parent process are under the same thread group (TG) umbrella, having the same TGID as the main thread but different <code class="language-plaintext highlighter-rouge">tid</code>s. <ul> <li>The main thread is the “process” that spawned all the children threads and has the same TGID as its PID.</li> </ul> </li> <li><code class="language-plaintext highlighter-rouge">/proc/[pid]/task/[tid]</code> shares the <strong>same</strong> content as <code class="language-plaintext highlighter-rouge">/proc/[pid]/</code> if <code class="language-plaintext highlighter-rouge">pid==tid</code> , i.e., it contains the same information describing the same process/thread.</li> <li>Therefore, when you look into the <code class="language-plaintext highlighter-rouge">/proc/[pid]</code> directory of a <em>multithreaded</em> process: <ul> <li>The <code class="language-plaintext highlighter-rouge">task/[tid]</code> subdirectories are all the threads within the same thread group.</li> <li>They have the same TGID being the <code class="language-plaintext highlighter-rouge">pid</code>.</li> <li>The so-called process is the main thread that spawned all other threads.</li> <li>The main thread has the same TGID as its PID (usually the smallest one under the <code class="language-plaintext highlighter-rouge">task/</code> directory).</li> </ul> </li> </ol> <h2 id="last-note-multiprocessing-vs-multithreading">Last Note: Multiprocessing vs. Multithreading</h2> <ol class="text-left"> <li>To avoid confusion, apart from the thread group ID (TGID), there is also a process group ID (PGID).</li> <li>The former is for multithreading, and the later is used in multiprocessing.</li> <li>A process is spawned by invoking the <code class="language-plaintext highlighter-rouge">fork()</code> syscall, a thread is created by e.g., <code class="language-plaintext highlighter-rouge">pthread_create()</code> in C. (Under the hood, they all use the syscall <code class="language-plaintext highlighter-rouge">clone()</code> but with different parameters) <ul> <li>The process that spawns other subprocesses is the <strong>parent</strong>. <ul> <li>When spawning a new process, the parent can either create a new process group (and puts itself and its children into it) or inherits that of an existing process (usually it’s the grandparent process).</li> <li>If a new process group is created, the PGID will be the PID of the process that creates it.</li> <li>If it’s inherited, the PGID is the same as that of the grandparent.</li> </ul> </li> <li>The process (the main thread) and all the threads it creates are <strong>siblings</strong>. <ul> <li>They share the same TGID.</li> <li>They also have the same PGID as that of the main thread.</li> <li><strong>NB</strong>: The threads created by a process are NOT visible to its parent process.</li> </ul> </li> </ul> </li> </ol> <h2 id="example">Example</h2> <ol class="text-left"> <li>We easily can launch a multithreaded program with <a href="https://wiki.ubuntu.com/Kernel/Reference/stress-ng"><code class="language-plaintext highlighter-rouge">stress-ng</code></a> on Linux.</li> <li>A “stressor/hog” is a process.</li> <li> <p>We can run the stress test for memory accesses with the following command, which will spawn a stressor that <em>runs with 5 threads</em> reading and writing to two different mappings of the same underlying physical page.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:~<span class="nv">$</span>stress-ng <span class="nt">--mcontend</span> 1 <span class="nt">-t</span> 10h stress-ng: info: <span class="o">[</span>56472] dispatching hogs: 1 mcontend </code></pre></div> </div> </li> <li> <p>With <code class="language-plaintext highlighter-rouge">htop</code>, we can see the process and threads therein in hierarchy (the <code class="language-plaintext highlighter-rouge">PGRP</code> is the GPID, and the <code class="language-plaintext highlighter-rouge">PID</code> is the ID for threads/processes).</p> <p><img src="http://hongyu.nl/_resources/pid.png" alt="htop results" /></p> <ul> <li>We can see that all the threads and processes have a unique ID, <code class="language-plaintext highlighter-rouge">PID</code> (process/thread ID). <ul> <li><code class="language-plaintext highlighter-rouge">56472</code>: The single-threaded <strong>parent</strong> process spawned from the bash command.</li> <li><code class="language-plaintext highlighter-rouge">56473</code>: The multithreaded <strong>child</strong> process (and the main thread) spawned from the parent <code class="language-plaintext highlighter-rouge">56472</code>.</li> <li><code class="language-plaintext highlighter-rouge">56473-56477</code>: The 5 <strong>sibling</strong> threads created by the main thread <code class="language-plaintext highlighter-rouge">56473</code>.</li> </ul> </li> </ul> </li> <li> <p>Then, by using <code class="language-plaintext highlighter-rouge">pidof</code>, we get the <code class="language-plaintext highlighter-rouge">pid</code> of the main thread.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:~<span class="nv">$ </span>pidof stress-ng-mcontend 56473 </code></pre></div> </div> </li> <li> <p>Since the bash command <code class="language-plaintext highlighter-rouge">56472</code> is the parent process that spawned the child process <code class="language-plaintext highlighter-rouge">56473</code>, we can examine this relationship by checking:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:~<span class="nv">$</span><span class="nb">cat</span> /proc/56472/task/56472/children 56473 </code></pre></div> </div> </li> <li> <p>Then, navigate to the <code class="language-plaintext highlighter-rouge">/proc</code> directory and check the <code class="language-plaintext highlighter-rouge">/proc/[pid]/task/</code> subdirectories. We get the 5 threads within this process:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:~<span class="nv">$ </span>ll /proc/56473/task/ total 0 dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 ./ dr-xr-xr-x 9 hy hy 0 Dec 31 22:21 ../ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56473/ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56474/ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56475/ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56476/ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56477/ </code></pre></div> </div> </li> <li> <p>To examine directory of a child thread, we get the same output as above since they are siblings.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:~<span class="nv">$</span>ll /proc/56476/task/ total 0 dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 ./ dr-xr-xr-x 9 hy hy 0 Dec 31 22:21 ../ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56473/ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56474/ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56475/ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56476/ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56477/ </code></pre></div> </div> </li> <li> <p>Note that the parent process <code class="language-plaintext highlighter-rouge">56472</code> is a single-threaded process, so its <code class="language-plaintext highlighter-rouge">/task</code> directory contains only itself.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:~<span class="nv">$ </span>ll /proc/56472/task/ total 0 dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 ./ dr-xr-xr-x 9 hy hy 0 Dec 31 22:21 ../ dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56472/ </code></pre></div> </div> </li> </ol> <h3 id="resource-accounting">Resource Accounting</h3> <ol class="text-left"> <li>The resource usage of a process/thread can be obtained in various ways, and the information is not consistent.</li> <li>From above, <code class="language-plaintext highlighter-rouge">htop</code> aggregate all the resource usages in the main thread <code class="language-plaintext highlighter-rouge">56473</code> by default.</li> <li>We can obtain per-process information from <code class="language-plaintext highlighter-rouge">ps</code> as well: <ul> <li> <p>Like <code class="language-plaintext highlighter-rouge">htop</code>, It <strong>aggregates</strong> usage of all threads to the main threads by default.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:~<span class="nv">$</span>ps <span class="nt">-p</span> 56473 <span class="nt">-o</span> %cpu,%mem,cmd %CPU %MEM CMD 473 0.0 stress-ng-mcontend </code></pre></div> </div> </li> <li> <p>It <em>only</em> works on the main thread but not with the siblings:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:~<span class="nv">$ </span>ps <span class="nt">-p</span> 56476 <span class="nt">-o</span> %cpu,%mem,cmd %CPU %MEM CMD </code></pre></div> </div> </li> <li> <p>To see detailed thread-level information, we can use the <code class="language-plaintext highlighter-rouge">-L</code> flag <em>on</em> <em>the main thread</em>:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">hy</span><span class="o">@</span><span class="n">node</span><span class="o">-</span><span class="mi">0</span><span class="p">:</span><span class="o">~</span><span class="err">$</span> <span class="n">ps</span> <span class="o">-</span><span class="n">L</span> <span class="mi">56473</span> <span class="o">-</span><span class="n">o</span> <span class="o">%</span><span class="n">cpu</span><span class="p">,</span><span class="o">%</span><span class="n">mem</span><span class="p">,</span><span class="n">cmd</span> <span class="o">%</span><span class="n">CPU</span> <span class="o">%</span><span class="n">MEM</span> <span class="n">CMD</span> <span class="mf">97.3</span> <span class="mf">0.0</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> <span class="mf">94.0</span> <span class="mf">0.0</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> <span class="mf">94.0</span> <span class="mf">0.0</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> <span class="mf">94.0</span> <span class="mf">0.0</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> <span class="mf">94.0</span> <span class="mf">0.0</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> </code></pre></div> </div> </li> <li> <p>With <code class="language-plaintext highlighter-rouge">-F</code> option, we can obtain the full glory:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">hy</span><span class="o">@</span><span class="n">node</span><span class="o">-</span><span class="mi">0</span><span class="p">:</span><span class="o">~</span><span class="err">$</span> <span class="n">ps</span> <span class="o">-</span><span class="n">L</span> <span class="mi">56473</span> <span class="o">-</span><span class="n">F</span> <span class="n">UID</span> <span class="n">PID</span> <span class="n">PPID</span> <span class="n">LWP</span> <span class="n">C</span> <span class="n">NLWP</span> <span class="n">SZ</span> <span class="n">RSS</span> <span class="n">PSR</span> <span class="n">STIME</span> <span class="n">TTY</span> <span class="n">STAT</span> <span class="n">TIME</span> <span class="n">CMD</span> <span class="n">hy</span> <span class="mi">56473</span> <span class="mi">56472</span> <span class="mi">56473</span> <span class="mi">97</span> <span class="mi">5</span> <span class="mi">22792</span> <span class="mi">2604</span> <span class="mi">13</span> <span class="mi">08</span><span class="p">:</span><span class="mi">10</span> <span class="n">pts</span><span class="o">/</span><span class="mi">2</span> <span class="n">RLl</span><span class="o">+</span> <span class="mi">302</span><span class="p">:</span><span class="mi">30</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> <span class="n">hy</span> <span class="mi">56473</span> <span class="mi">56472</span> <span class="mi">56474</span> <span class="mi">94</span> <span class="mi">5</span> <span class="mi">22792</span> <span class="mi">2604</span> <span class="mi">7</span> <span class="mi">08</span><span class="p">:</span><span class="mi">10</span> <span class="n">pts</span><span class="o">/</span><span class="mi">2</span> <span class="n">RLl</span><span class="o">+</span> <span class="mi">292</span><span class="p">:</span><span class="mi">03</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> <span class="n">hy</span> <span class="mi">56473</span> <span class="mi">56472</span> <span class="mi">56475</span> <span class="mi">94</span> <span class="mi">5</span> <span class="mi">22792</span> <span class="mi">2604</span> <span class="mi">31</span> <span class="mi">08</span><span class="p">:</span><span class="mi">10</span> <span class="n">pts</span><span class="o">/</span><span class="mi">2</span> <span class="n">RLl</span><span class="o">+</span> <span class="mi">292</span><span class="p">:</span><span class="mi">00</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> <span class="n">hy</span> <span class="mi">56473</span> <span class="mi">56472</span> <span class="mi">56476</span> <span class="mi">94</span> <span class="mi">5</span> <span class="mi">22792</span> <span class="mi">2604</span> <span class="mi">15</span> <span class="mi">08</span><span class="p">:</span><span class="mi">10</span> <span class="n">pts</span><span class="o">/</span><span class="mi">2</span> <span class="n">RLl</span><span class="o">+</span> <span class="mi">291</span><span class="p">:</span><span class="mi">59</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> <span class="n">hy</span> <span class="mi">56473</span> <span class="mi">56472</span> <span class="mi">56477</span> <span class="mi">94</span> <span class="mi">5</span> <span class="mi">22792</span> <span class="mi">2604</span> <span class="mi">0</span> <span class="mi">08</span><span class="p">:</span><span class="mi">10</span> <span class="n">pts</span><span class="o">/</span><span class="mi">2</span> <span class="n">RLl</span><span class="o">+</span> <span class="mi">292</span><span class="p">:</span><span class="mi">05</span> <span class="n">stress</span><span class="o">-</span><span class="n">ng</span><span class="o">-</span><span class="n">mcontend</span> </code></pre></div> </div> </li> <li> <p>Note that <strong>ALL</strong> threads share the same PID but each of them has a unique TID (<code class="language-plaintext highlighter-rouge">LWP</code>).</p> </li> </ul> </li> <li> <p>We can monitor those threads using <code class="language-plaintext highlighter-rouge">top</code> with <code class="language-plaintext highlighter-rouge">-H</code>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:/proc<span class="nv">$</span>top <span class="nt">-H</span> <span class="nt">-p</span> 56476 .... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 56473 hy 20 0 91168 2708 2272 R 97.3 0.0 127:24.91 stress-ng-mcont 56474 hy 20 0 91168 2708 2272 R 94.0 0.0 122:55.16 stress-ng-mcont 56475 hy 20 0 91168 2708 2272 R 94.0 0.0 122:54.44 stress-ng-mcont 56476 hy 20 0 91168 2708 2272 R 93.7 0.0 122:55.33 stress-ng-mcont 56477 hy 20 0 91168 2708 2272 R 92.3 0.0 122:56.57 stress-ng-mcont </code></pre></div> </div> <ul> <li> <p>Without the <code class="language-plaintext highlighter-rouge">-H</code> flag, however, it <strong>aggregates</strong> all usages to <strong>any</strong> of the sibling threads:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> hy@node-0:~<span class="nv">$ </span>top <span class="nt">-p</span> 56476 .... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 56473 hy 20 0 91168 2708 2272 R 476.3 0.0 621:39.36 stress-ng-mcont </code></pre></div> </div> </li> <li> <p>Also, we can get a single number out:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c"># Get the CPU utilization of thread 56476.</span> <span class="c"># * Aggregated.</span> hy@node-0:~<span class="nv">$</span>top <span class="nt">-b</span> <span class="nt">-n</span> 2 <span class="nt">-d</span> 0.2 <span class="nt">-p</span> 56476 | <span class="nb">tail</span> <span class="nt">-1</span> | <span class="nb">awk</span> <span class="s1">'{print$9}'</span> 465.0 <span class="c"># * Per-thread with -H.</span> hy@node-0:~<span class="nv">$</span>top <span class="nt">-b</span> <span class="nt">-H</span> <span class="nt">-n</span> 2 <span class="nt">-d</span> 0.2 <span class="nt">-p</span> 56476 | <span class="nb">tail</span> <span class="nt">-1</span> | <span class="nb">awk</span> <span class="s1">'{print$9}'</span> 75.0 </code></pre></div> </div> </li> </ul> </li> <li>How to get per-thread resource usages? We need to again look into the <code class="language-plaintext highlighter-rouge">/proc</code> sysfs. <ul> <li>The file <code class="language-plaintext highlighter-rouge">/proc/pid/stat</code> of a thread (whose TID==<code class="language-plaintext highlighter-rouge">pid</code>) <strong>contains the information aggregated from all threads</strong>.</li> <li> <p>The file <code class="language-plaintext highlighter-rouge">/proc/pid/task/pid/stat</code> contains per-thread information:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1"># * Get total cpu time (user and kernel) of all threads belonging to the same TG as that of 56476. </span> <span class="n">hy</span><span class="o">@</span><span class="n">node</span><span class="o">-</span><span class="mi">0</span><span class="p">:</span><span class="o">~</span><span class="err">$</span> <span class="n">cat</span> <span class="o">/</span><span class="n">proc</span><span class="o">/</span><span class="mi">56476</span><span class="o">/</span><span class="n">stat</span> <span class="o">|</span> <span class="n">awk</span> <span class="s">'{print$14, $15}'</span> <span class="mi">9460932</span> <span class="mi">12361</span> <span class="c1"># * Get the cpu time for only thread 56476. </span> <span class="n">hy</span><span class="o">@</span><span class="n">node</span><span class="o">-</span><span class="mi">0</span><span class="p">:</span><span class="o">~</span><span class="err">$</span> <span class="n">cat</span> <span class="o">/</span><span class="n">proc</span><span class="o">/</span><span class="mi">56476</span><span class="o">/</span><span class="n">task</span><span class="o">/</span><span class="mi">56476</span><span class="o">/</span><span class="n">stat</span> <span class="o">|</span> <span class="n">awk</span> <span class="s">'{print $14,$15}'</span> <span class="mi">1879429</span> <span class="mi">3032</span> </code></pre></div> </div> </li> </ul> </li> <li> <p>Alternatively, we can use the mighty python with <code class="language-plaintext highlighter-rouge">psutil</code>.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">psutil</span> <span class="c1"># The great grandparent process. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tmux_session</span> <span class="o">=</span> <span class="n">psutil</span><span class="p">.</span><span class="n">Process</span><span class="p">(</span><span class="mi">54711</span><span class="p">)</span> <span class="c1"># * It's spawned from the mother of all processes of PID=1 -- the init(old distros)/systemd(new distros). </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tmux_session</span><span class="p">.</span><span class="n">ppid</span><span class="p">()</span> <span class="mi">1</span> <span class="o">&gt;&gt;&gt;</span> <span class="p">[(</span><span class="n">child</span><span class="p">.</span><span class="n">name</span><span class="p">(),</span> <span class="n">child</span><span class="p">.</span><span class="n">pid</span><span class="p">)</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">tmux_session</span><span class="p">.</span><span class="n">children</span><span class="p">(</span><span class="n">recursive</span><span class="o">=</span><span class="bp">True</span><span class="p">)]</span> <span class="p">[(</span><span class="s">'bash'</span><span class="p">,</span> <span class="mi">54712</span><span class="p">),</span> <span class="p">(</span><span class="s">'bash'</span><span class="p">,</span> <span class="mi">56236</span><span class="p">),</span> <span class="p">(</span><span class="s">'python'</span><span class="p">,</span> <span class="mi">56613</span><span class="p">),</span> <span class="p">(</span><span class="s">'stress-ng'</span><span class="p">,</span> <span class="mi">56472</span><span class="p">),</span> <span class="p">(</span><span class="s">'stress-ng-mcontend'</span><span class="p">,</span> <span class="mi">56473</span><span class="p">)]</span> <span class="c1"># The parent process. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span> <span class="o">=</span> <span class="n">psutil</span><span class="p">.</span><span class="n">Process</span><span class="p">(</span><span class="mi">56472</span><span class="p">)</span> <span class="c1"># * The parent was spawned from one of the above bash sessions (grandparent) </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span><span class="p">.</span><span class="n">ppid</span><span class="p">()</span> <span class="mi">54712</span> <span class="c1"># * The sibling threads art NOT children, and invisable to the parent process. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span><span class="p">.</span><span class="n">children</span><span class="p">(</span><span class="n">recursive</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="p">[</span><span class="n">psutil</span><span class="p">.</span><span class="n">Process</span><span class="p">(</span><span class="n">pid</span><span class="o">=</span><span class="mi">56473</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">'stress-ng-mcontend'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="s">'running'</span><span class="p">,</span> <span class="n">started</span><span class="o">=</span><span class="s">'11:21:57'</span><span class="p">)]</span> <span class="c1"># * The parent is single-threaded. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span><span class="p">.</span><span class="n">num_threads</span><span class="p">()</span> <span class="mi">1</span> <span class="c1"># The child process spawned from the parent process. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">child</span> <span class="o">=</span> <span class="n">psutil</span><span class="p">.</span><span class="n">Process</span><span class="p">(</span><span class="mi">56473</span><span class="p">)</span> <span class="c1"># * The child created 5 threads (including itself). </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">child</span><span class="p">.</span><span class="n">num_threads</span><span class="p">()</span> <span class="mi">5</span> <span class="o">&gt;&gt;&gt;</span> <span class="p">[</span><span class="n">thread</span><span class="p">.</span><span class="nb">id</span> <span class="k">for</span> <span class="n">thread</span> <span class="ow">in</span> <span class="n">child</span><span class="p">.</span><span class="n">threads</span><span class="p">()]</span> <span class="p">[</span><span class="mi">56473</span><span class="p">,</span> <span class="mi">56474</span><span class="p">,</span> <span class="mi">56475</span><span class="p">,</span> <span class="mi">56476</span><span class="p">,</span> <span class="mi">56477</span><span class="p">]</span> <span class="c1"># One of the sibling threads. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">sibling</span> <span class="o">=</span> <span class="n">psutil</span><span class="p">.</span><span class="n">Process</span><span class="p">(</span><span class="mi">56476</span><span class="p">)</span> <span class="c1"># * The sibling thread inherits the parent process of the main thread. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">child</span><span class="p">.</span><span class="n">ppid</span><span class="p">()</span> <span class="mi">56472</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">sibling</span><span class="p">.</span><span class="n">ppid</span><span class="p">()</span> <span class="mi">56472</span> <span class="c1"># Accounting resouces. </span> <span class="c1"># * The parent process is in sleep state (S), so it doesn't take any CPU time. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span><span class="p">.</span><span class="n">cpu_percent</span><span class="p">(</span><span class="n">interval</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="mf">0.0</span> <span class="c1"># ! psutil **aggregates** all sibling resources to **any** of the siblings. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">sibling</span><span class="p">.</span><span class="n">cpu_percent</span><span class="p">(</span><span class="n">interval</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="mf">471.5</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">child</span><span class="p">.</span><span class="n">cpu_percent</span><span class="p">(</span><span class="n">interval</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="mf">472.4</span> <span class="c1"># ! Also, its cpu time accounting for children processes is broken somehow ... </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tmux_session</span><span class="p">.</span><span class="n">cpu_times</span><span class="p">()</span> <span class="n">pcputimes</span><span class="p">(</span><span class="n">user</span><span class="o">=</span><span class="mf">7.46</span><span class="p">,</span> <span class="n">system</span><span class="o">=</span><span class="mf">3.19</span><span class="p">,</span> <span class="n">children_user</span><span class="o">=</span><span class="mf">102.18</span><span class="p">,</span> <span class="n">children_system</span><span class="o">=</span><span class="mf">153.15</span><span class="p">,</span> <span class="n">iowait</span><span class="o">=</span><span class="mf">0.0</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span><span class="p">.</span><span class="n">cpu_times</span><span class="p">()</span> <span class="n">pcputimes</span><span class="p">(</span><span class="n">user</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">system</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">children_user</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">children_system</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">iowait</span><span class="o">=</span><span class="mf">0.0</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">child</span><span class="p">.</span><span class="n">cpu_times</span><span class="p">()</span> <span class="n">pcputimes</span><span class="p">(</span><span class="n">user</span><span class="o">=</span><span class="mf">45250.11</span><span class="p">,</span> <span class="n">system</span><span class="o">=</span><span class="mf">57.79</span><span class="p">,</span> <span class="n">children_user</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">children_system</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">iowait</span><span class="o">=</span><span class="mf">0.0</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">sibling</span><span class="p">.</span><span class="n">cpu_times</span><span class="p">()</span> <span class="n">pcputimes</span><span class="p">(</span><span class="n">user</span><span class="o">=</span><span class="mf">45255.42</span><span class="p">,</span> <span class="n">system</span><span class="o">=</span><span class="mf">57.79</span><span class="p">,</span> <span class="n">children_user</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">children_system</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">iowait</span><span class="o">=</span><span class="mf">0.0</span><span class="p">)</span> <span class="c1"># * Memory usages are accounted the same as it does for CPU. </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span><span class="p">.</span><span class="n">memory_full_info</span><span class="p">()</span> <span class="n">pfullmem</span><span class="p">(</span><span class="n">rss</span><span class="o">=</span><span class="mi">6475776</span><span class="p">,</span> <span class="n">vms</span><span class="o">=</span><span class="mi">59777024</span><span class="p">,</span> <span class="n">shared</span><span class="o">=</span><span class="mi">6078464</span><span class="p">,</span> <span class="n">text</span><span class="o">=</span><span class="mi">1728512</span><span class="p">,</span> <span class="n">lib</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="mi">32018432</span><span class="p">,</span> <span class="n">dirty</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">uss</span><span class="o">=</span><span class="mi">3051520</span><span class="p">,</span> <span class="n">pss</span><span class="o">=</span><span class="mi">3749888</span><span class="p">,</span> <span class="n">swap</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">child</span><span class="p">.</span><span class="n">memory_full_info</span><span class="p">()</span> <span class="n">pfullmem</span><span class="p">(</span><span class="n">rss</span><span class="o">=</span><span class="mi">2772992</span><span class="p">,</span> <span class="n">vms</span><span class="o">=</span><span class="mi">93356032</span><span class="p">,</span> <span class="n">shared</span><span class="o">=</span><span class="mi">2326528</span><span class="p">,</span> <span class="n">text</span><span class="o">=</span><span class="mi">1728512</span><span class="p">,</span> <span class="n">lib</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="mi">65581056</span><span class="p">,</span> <span class="n">dirty</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">uss</span><span class="o">=</span><span class="mi">126976</span><span class="p">,</span> <span class="n">pss</span><span class="o">=</span><span class="mi">735232</span><span class="p">,</span> <span class="n">swap</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">sibling</span><span class="p">.</span><span class="n">memory_full_info</span><span class="p">()</span> <span class="n">pfullmem</span><span class="p">(</span><span class="n">rss</span><span class="o">=</span><span class="mi">2772992</span><span class="p">,</span> <span class="n">vms</span><span class="o">=</span><span class="mi">93356032</span><span class="p">,</span> <span class="n">shared</span><span class="o">=</span><span class="mi">2326528</span><span class="p">,</span> <span class="n">text</span><span class="o">=</span><span class="mi">1728512</span><span class="p">,</span> <span class="n">lib</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="mi">65581056</span><span class="p">,</span> <span class="n">dirty</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">uss</span><span class="o">=</span><span class="mi">126976</span><span class="p">,</span> <span class="n">pss</span><span class="o">=</span><span class="mi">735232</span><span class="p">,</span> <span class="n">swap</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">tmux_session</span><span class="p">.</span><span class="n">memory_percent</span><span class="p">()</span> <span class="mf">0.007239506814671662</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span><span class="p">.</span><span class="n">memory_percent</span><span class="p">()</span> <span class="mf">0.009602063988251591</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">child</span><span class="p">.</span><span class="n">memory_percent</span><span class="p">()</span> <span class="mf">0.004111699759675096</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">sibling</span><span class="p">.</span><span class="n">memory_percent</span><span class="p">()</span> <span class="mf">0.004111699759675096</span> </code></pre></div> </div> </li> <li>In summary, there are various ways of accounting resources for processes and threads on Linux. <ul> <li>However, there are nuances that must be taken into account.</li> <li><code class="language-plaintext highlighter-rouge">psutil</code> is a convenient tool for sys admins when scripting in python. <ul> <li>However, the resource usage of <strong>any</strong> threads is the <strong>aggregated</strong> values of all threads!</li> <li>It doesn’t explicitly distinguish between PIDs and TIDs either.</li> </ul> </li> <li> <p>And it’s time to end our running example:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1"># ! "Note this will return True also if the process is a zombie (p.status() == psutil.STATUS_ZOMBIE)" </span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span><span class="p">.</span><span class="n">is_running</span><span class="p">()</span> <span class="o">==</span> <span class="n">child</span><span class="p">.</span><span class="n">is_running</span><span class="p">()</span> <span class="o">==</span> <span class="n">sibling</span><span class="p">.</span><span class="n">is_running</span><span class="p">()</span> <span class="o">==</span> <span class="bp">True</span> <span class="bp">True</span> <span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">signal</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">sibling</span><span class="p">.</span><span class="n">send_signal</span><span class="p">(</span><span class="n">signal</span><span class="p">.</span><span class="n">SIGINT</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">parent</span><span class="p">.</span><span class="n">is_running</span><span class="p">()</span> <span class="o">==</span> <span class="n">child</span><span class="p">.</span><span class="n">is_running</span><span class="p">()</span> <span class="o">==</span> <span class="n">sibling</span><span class="p">.</span><span class="n">is_running</span><span class="p">()</span> <span class="o">==</span> <span class="bp">False</span> <span class="bp">True</span> </code></pre></div> </div> <p>(It seems that interrupting one thread has a bottom-up cascading effect in <code class="language-plaintext highlighter-rouge">stress-ng</code> 🥴 )</p> </li> </ul> </li> </ol> <p class="text-left">Happy New Year 🎆 ~</p> <hr /> <h1 id="reference">Reference</h1> <ul> <li><a href="https://unix.stackexchange.com/questions/364660/are-threads-implemented-as-processes-on-linux">https://unix.stackexchange.com/questions/364660/are-threads-implemented-as-processes-on-linux</a></li> <li><a href="https://unix.stackexchange.com/questions/670836/why-do-threads-have-their-own-pid">https://unix.stackexchange.com/questions/670836/why-do-threads-have-their-own-pid</a></li> <li><a href="https://stackoverflow.com/questions/1221555/retrieve-cpu-usage-and-memory-usage-of-a-single-process-on-linux">https://stackoverflow.com/questions/1221555/retrieve-cpu-usage-and-memory-usage-of-a-single-process-on-linux</a></li> <li><a href="https://unix.stackexchange.com/questions/404054/how-is-a-process-group-id-set">https://unix.stackexchange.com/questions/404054/how-is-a-process-group-id-set</a></li> <li><a href="https://stackoverflow.com/questions/4856255/the-difference-between-fork-vfork-exec-and-clone">https://stackoverflow.com/questions/4856255/the-difference-between-fork-vfork-exec-and-clone</a></li> <li><a href="https://stackoverflow.com/questions/19678954/relation-between-thread-id-and-process-id">https://stackoverflow.com/questions/19678954/relation-between-thread-id-and-process-id</a></li> <li><a href="https://stackoverflow.com/questions/9430491/find-cpu-usage-for-a-thread-in-linux">https://stackoverflow.com/questions/9430491/find-cpu-usage-for-a-thread-in-linux</a></li> <li><a href="https://stackoverflow.com/questions/1420426/how-to-calculate-the-cpu-usage-of-a-process-by-pid-in-linux-from-c">https://stackoverflow.com/questions/1420426/how-to-calculate-the-cpu-usage-of-a-process-by-pid-in-linux-from-c</a></li> <li><a href="https://stackoverflow.com/questions/19919881/sysconf-sc-clk-tck-what-does-it-return">https://stackoverflow.com/questions/19919881/sysconf-sc-clk-tck-what-does-it-return</a></li> <li><a href="https://www.baeldung.com/linux/total-process-cpu-usage">https://www.baeldung.com/linux/total-process-cpu-usage</a></li> <li><a href="https://psutil.readthedocs.io/en/latest/#processes">https://psutil.readthedocs.io/en/latest/#processes</a></li> <li><a href="https://man7.org/linux/man-pages/man5/proc.5.html#top_of_page">https://man7.org/linux/man-pages/man5/proc.5.html</a></li> <li><a href="https://man7.org/linux/man-pages/man2/getpid.2.html">https://man7.org/linux/man-pages/man2/getpid.2.html</a></li> <li><a href="https://man7.org/linux/man-pages/man2/gettid.2.html">https://man7.org/linux/man-pages/man2/gettid.2.html</a></li> <li><a href="https://manpages.ubuntu.com/manpages/bionic/man1/stress-ng.1.html">https://manpages.ubuntu.com/manpages/bionic/man1/stress-ng.1.html</a></li> <li><a href="https://www.akkadia.org/drepper/nptl-design.pdf">https://www.akkadia.org/drepper/nptl-design.pdf</a></li> </ul>Hongyu Hèsystems basicsConnecting Bayesian with Regularization2022-01-03T00:00:00+00:002022-01-03T00:00:00+00:00http://hongyu.nl/baysian<h2 id="bias-variance-trade-off">Bias-Variance Trade-off</h2> <p>In many cases, it’s desired to trade off a bit larger bias for a much smaller variance for better estimations of the generalisation error. One way to do so is by adding a regularizer. In the case of linear regression, L1 (Lasso) and L2 (Ridge) regularization are two of the most common ones. Lasso suppresses weights of small magnitudes to zero, making the feature space sparse, whilst Ridge “condenses” all weights to smaller values. Both can restrict the norm of the weights and therefore, mitigate overfitting.</p> <p>However, regularization is only one way to strike the balance. Another way is to introduce more bias into the equation through the Bayesian lens. Specifically, we can impose prior knowledge by adding a prior distribution to constrain the norm of the leant parameters. For example, if we know the weights are small and centred, then we can set our prior to be $\vec{w} \sim \mathcal{N}(0, \beta\textbf{I})$. Then, by Bayes rule, we have:</p> $\mathbb{P}(\vec{w} | X, \vec{y}) = \cfrac{\mathbb{P}(\vec{w}, X, \vec{y})}{\mathbb{P}(X, \vec{y})} = \cfrac{\mathbb{P}(\vec{w}, \vec{y} | X) {\mathbb{P}(X)}}{\mathbb{P}(\vec{y} | X) {\mathbb{P}(X)}} = \cfrac{\mathbb{P}(\vec{w}, \vec{y} | X)}{\mathbb{P}(\vec{y} | X)}$ <p>Thus, both regularization and Bayesian modelling can achieve the same goal, which begs the question: are they connected?</p> <h2 id="laplace-is-to-lasso-as-gaussian-is-to-ridge">Laplace is to Lasso as Gaussian is to Ridge</h2> <p>The answer to the above question turns out to be yes! To illustrate this further, let’s use two common prior distributions, Lapace and Gaussian, as our running examples.</p> <p>Firstly, we assume the following general setting for regression: ${y} = X{\theta}$ and $f_X = y + \epsilon$ where $\theta \sim \text{Laplace}(0, s) = 1/2s \cdot\exp(-\mid\theta\mid / s)$ and $\epsilon \sim \mathcal{N}(0, \delta^2_\epsilon)$.</p> <p>Then, we obtain the maximum a posteriori (MAP) esitmation as:</p> \begin{align} \arg\max_\theta\mathbb{P}({\theta} | X, {y}) &amp;= \arg\max_\theta\cfrac{\mathbb{P}(y | X, \theta)\mathbb{P}(\theta)} {\mathbb{P}(y)} \nonumber\\ &amp;\propto \arg\max_\theta\mathbb{P}(y |X, \theta)\mathbb{P}(X| \theta)\mathbb{P}(\theta) \nonumber\\ &amp;\propto \arg\max_\theta\mathbb{P}(y | X, \theta)\mathbb{P}(\theta) \nonumber\\ &amp;\propto \arg\max_\theta\mathbb{P}(\theta) \prod^n_i \mathbb{P}(y_i | X_i, \theta) \nonumber\\ &amp;\propto \arg\min_\theta -\log \mathbb{P}(\theta) - \sum_i^n \log \mathbb{P}_\theta(y_i | X_i) \end{align} <p>Next, we can substitute both the likelihood and prior into Eq. (1).</p> \begin{align} \arg\min_\theta -\log\cfrac{1}{2s} \exp\left\{-\cfrac{|\theta|}{s}\right\} - \sum^n_i \log \cfrac{1}{Z} \exp\left\{-\cfrac{1}{2}\left(\cfrac{y_i - f_i}{\delta_\epsilon}\right)^2\right\} \end{align} <p>where $Z$ is the Gaussian normalising constant. By simplifying Eq. (2), we obtain the following form:</p> \begin{align} &amp; \arg\min_\theta - \cfrac{|\theta|}{s} + \cfrac{1}{2\delta^2_\epsilon} \sum^n_i(y_i - f_i)^2 \\ =&amp; \arg\min_\theta - \sum^n_i(y_i - f_i)^2 - \cfrac{2\delta^2_\epsilon}{s}||\theta||_1 \end{align} <p>Now, we have recovered the exact form of Lasso, where $\cfrac{2\delta^2_\epsilon}{s}$ is the coefficient of the L1 regularizor $\lambda$ that controls the strength of the constraint.</p> <hr /> <p>Next, let’s play the same trick in the same setting but with a Gaussian prior instead, i.e., $\theta \sim \mathcal{N}(0, \delta_\theta^2)$.</p> <p>Starting from Eq. (1), we subsitute in the likelihood and prior as above:</p> \begin{align} \arg\max_\theta\mathbb{P}({\theta} | X, {y}) &amp;\propto \arg\min_\theta -\log \mathbb{P}(\theta) - \sum_i^n \log \mathbb{P}_\theta(y_i | X_i) \nonumber \\ &amp;\propto \arg\min_\theta -\log \cfrac{1}{Z'} \exp\left\{-\cfrac{1}{2}\left(\cfrac{\theta-0}{\delta_\theta}\right)^2 \right\} \\ &amp; \quad - \sum^n_i \log \cfrac{1}{Z} \exp\left\{-\cfrac{1}{2}\left(\cfrac{y_i - f_i}{\delta_\epsilon}\right)^2\right\} \nonumber \end{align} <p>Finally, letting go all the fluff in Eq. (5), we have:</p> \begin{align} &amp; \arg\min_\theta - \cfrac{||\theta||_2^2}{2\delta^2_\theta} + \cfrac{1}{2\delta^2_\epsilon} \sum^n_i(y_i - f_i)^2 \nonumber\\ =&amp; \arg\min_\theta - \sum^n_i(y_i - f_i)^2 - \cfrac{\delta^2_\epsilon}{\delta^2_\theta}||\theta||_2^2 \end{align} <p>By Eq. (6), we have recovered Ridge regression where the fraction $\cfrac{\delta_{\epsilon}^2}{\delta_{\theta}^2}$ denotes regularization constant $\lambda$.</p> <h2 id="summary">Summary</h2> <p>By working out the above two examples, we found that regularised regression is nothing but Bayesian modelling in disguise. In fact, <em>imposing various priors has the same effect as using corresponding regularizers.</em> By the same token, <em>choosing different likelihoods gives us different loss functions.</em> In this post, we used Gaussian likelihood in both examples, and, in turn, recovered the square loss.</p> <p>There are many other options for prior and likelihood functions. For instance, one can use a student-t as opposed to Gaussian. Lastly, a family of <a href="https://en.wikipedia.org/wiki/Conjugate_prior">conjugate prior</a> can drastically reduce the cost of Bayesian inference.</p>Hongyu Hèmachine learningGood Reads2021-10-08T00:00:00+00:002021-10-08T00:00:00+00:00http://hongyu.nl/books<p class="notice">This’s also a rolling log in which I list some books recently read (2021) that are right up to my alley.</p> <h3 class="text-justify" id="bennett-arnold-self-and-self-management-essays-about-existing-george-h-doran-company-1918">Bennett, Arnold. <em>Self and self-management: Essays about existing.</em> George H. Doran Company, 1918.</h3> <p>The true stories behind the stories of success shall be the same and might not be as glorious.</p> <h3 class="text-justify" id="hawking-stephen-the-theory-of-everything-jaico-publishing-house-2006">Hawking, Stephen. <em>The theory of everything.</em> Jaico Publishing House, 2006.</h3> <p>Well, I’m now under the impression that Computer Science is a pseudoscience :}</p> <h3 class="text-justify" id="hall-herbert-james-the-untroubled-mind-houghton-mifflin-1915">Hall, Herbert James. <em>The untroubled mind.</em> Houghton Mifflin, 1915.</h3> <p>How can I live out a life so fully that worries couldn’t sneak in? Perhaps most importantly, what’s my deeper justification and higher pursuit thereof?</p> <h3 class="text-justify" id="jones-diana-wynne-john-sessions-and-stella-paskins-howls-moving-castle-recorded-books-2008">Jones, Diana Wynne, John Sessions, and Stella Paskins. <em>Howl’s moving castle.</em> Recorded Books, 2008.</h3> <p>I finally understand why I didn’t quite understand the movie 😅</p>Hongyu Hè📚My Take on TED Talks2021-08-19T00:00:00+00:002021-08-19T00:00:00+00:00http://hongyu.nl/ted<p>For the past several months, I’ve picked up the habit of watching TED talks. However, I found that, even if it’s an impressive speech with resonating ideas, it won’t take long before it vanishes from my memory. This fretted me, so I decided to start briefly jotting down my main takeaways in this rolling log.</p> <p class="notice">Some of my friends from China saw a bunch of blanks here. This is because YouTube videos can’t pass the firewall 🧱.</p> <h2 id="10-ways-to-have-a-better-conversation">10 ways to have a better conversation</h2> <iframe width="560" height="315" src="https://www.youtube.com/embed/R1vskiVDwl4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <ul> <li>If you want to pontificate, go write a blog.</li> <li>Ask open-ended questions.</li> <li>“No man ever listened his way out of a job” — Calvin Coolidge</li> <li>Most of us don’t listen with the intent to understand. We listen with the intent to reply.</li> <li>Listen, and be prepared to be amazed.</li> <li>If you don’t know, say that you don’t know.</li> <li>Don’t equate your experience with others.</li> </ul> <h2 id="self-control">Self-control</h2> <iframe width="560" height="315" src="https://www.youtube.com/embed/PPQhj6ktYSo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <p class="notice--warning">This Ted talk conveys very similar messages as that of the below one. Their main ideas are covered in many other talks as well.</p> <h2 id="to-reach-beyond-your-limits-by-training-your-mind">To reach beyond your limits by training your mind</h2> <iframe width="560" height="315" src="https://www.youtube.com/embed/zCv-ZBy6_yU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <ul> <li>Self-control is the problem where we have all these desires from ourselves for the long-term, but then in the short-term, we do rather different things (that prevent ourselves from achieving long-term those goals).</li> <li>Our will power is weak, and therefore, it’s not something upon which we should rely during decision-making.</li> <li>If we are faced with temptation whilst having no tool at hand to overcome it, we’re almost certainly going to fail.</li> <li>We should collaborate with our brains with constructive messages. <ul> <li>Change the pictures and the words. Using very detailed words.</li> <li>Tell your mind exactly what you want.</li> </ul> </li> <li>We ought to create tools that will control our future selves to do what our current selves want them to do. <ul> <li>It’s a situation where we know we will be tempted, and we do something to make ourselves not be able to be tempted.</li> </ul> </li> <li>Make the familiar unfamiliar and the unfamiliar familiar.</li> <li>Reward-substitution: connecting pain to pleasure <ul> <li>Link massive pleasure to going there and pain to not going there.</li> <li>Do the right thing for the wrong reason.</li> </ul> </li> <li>Make Self Belief so normal to you that everyone believes in you too.</li> </ul> <p class="notice--info">The last point ties in with another Ted talk, which will be introduced later.</p> <h2 id="body-language-the-power-is-in-the-palm-of-your-hands">Body language, the power is in the palm of your hands</h2> <iframe width="560" height="315" src="https://www.youtube.com/embed/ZZZ7k8cMA-4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <ul> <li>The palm up position is friendly and inviting, whilst the palm down position exerts power and control over others. <ul> <li>Finger pointing is the worst; it is directive and rude.</li> </ul> </li> </ul> <h2 id="imposters-the-psychology-of-pretending-to-be-someone-youre-not">Imposters: The psychology of pretending to be someone you’re not</h2> <iframe width="560" height="315" src="https://www.youtube.com/embed/vSjlCJaEwZE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <ul> <li>Some imposters are escapologists — they’re running away from a flawed past and trying to rehabilitate their image.</li> <li>We spend a hell of a lot of time trying to impress other people.</li> <li>Ultimately, life is a performance with all the sense of drama, anarchy, and possibility.</li> <li>Who ever controls the past controls the future; who ever controls the present controls the past. — Orwell</li> </ul> <h2 id="how-to-draw-to-remember-more">How to draw to remember more</h2> <iframe width="560" height="315" src="https://www.youtube.com/embed/gj3ZnKlHqxI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <ul> <li>Thinking in pictures and then drawing them down is a great way to learn and memorize new things.</li> <li>There are a myriad of ways to represent an abstract concept using drawings.</li> <li>The quality of the drawing does not matter at all. In other words, good and bad pictures have rather similar, if not the same, effect on the learning process.</li> <li>It is the process of doing the drawing that actually makes a difference.</li> </ul> <p class="notice--success">To be continued 👨‍💻 …</p>Hongyu Hèwho's Ted?SSH Proxy Jump2020-08-11T00:00:00+00:002020-08-11T00:00:00+00:00http://hongyu.nl/ssh<p>This post is concerned with the basics of SSH authentication, as well as its indirect login via a proxy server.</p> <figure style="width: 90%" class="align-center"> <img src="http://hongyu.nl/assets/images/ssh.png" alt="" /> <!-- <figcaption>System overview</figcaption> --> </figure> <h2 id="step-1-key-pair-generation">Step 1: key pair generation</h2> <p>Use the following command to generate a public(silver)/private(black) RSA key pair under the <code class="language-plaintext highlighter-rouge">~/.ssh/id_rsa</code> directory. The <code class="language-plaintext highlighter-rouge">.ssh/id_rsa</code> is the private key that you keep in your machine, and the <code class="language-plaintext highlighter-rouge">.ssh/id_rsa.pub</code> is the public key that you distribute to other machines/platforms in order to achieve automatic login via key-pair verification.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh-keygen <span class="nt">-t</span> rsa </code></pre></div></div> <p class="notice--info">Note that git uses a different type of cryptosystem, namely, the Ed25519 system, which can be generated using the following command.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh-keygen <span class="nt">-t</span> ed25519 <span class="nt">-C</span> <span class="s2">"youremail@yourdomain"</span> </code></pre></div></div> <p>Compared to RSA, it is considered to be faster, safer and more compact (Ed25519: 8chars, RSA: 544chars) although RSA is more commonly used.</p> <h2 id="step-2-distribute-keys">Step 2: distribute key(s)</h2> <p>If <em>Alice</em> wants to log in <em>Server1</em> shown in the figure, she can run the following command to forward her SSH <strong>public key(s)</strong> to it. The keys sent will be recorded in <code class="language-plaintext highlighter-rouge">.ssh/authorized_keys</code> of the host.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh-copy-id alice-username@server1.domain-or-ip </code></pre></div></div> <p>Afterwards, <em>Alice</em> should be able to log in <em>Server1</em> without being asked for her password. I.e.,</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh alice-username@server1.domain-or-ip Welcome to XXX ... </code></pre></div></div> <p class="notice--warning">Note that for Mac users that do not have <code class="language-plaintext highlighter-rouge">ssh-copy-id</code>, you can either intall it via <code class="language-plaintext highlighter-rouge">brew</code> or mannually copy these ssh files through <code class="language-plaintext highlighter-rouge">scp</code>, <code class="language-plaintext highlighter-rouge">rsync</code> or whatnot. If you go for the latter, one thing you should keep in mind is to set the permission bits correctly as shown below.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">chmod </span>700 ~/.ssh <span class="nb">chmod </span>600 ~/.ssh/<span class="k">*</span> </code></pre></div></div> <h2 id="step-3-indirect-login">Step 3: indirect login</h2> <p>In the case that <em>Server1</em> has a firewall or what have you, <em>Alice</em> has to connect it via a proxy, say <em>Server2</em>; therein lies the question: how to access <em>Server2</em> directly using key-pair authentication?</p> <p>To tackle this, <em>Alice</em> can first forward her/his keys to <em>Server2</em>, the proxy, through <strong>Step 2</strong>. Next, <em>Alice</em> should log in <em>Server2</em> to generate a key pair (<strong>Step 1</strong>), and then, send keys to <em>Server1</em>.</p> <p>Last, but certainly not least, <em>Alice</em> should set up her <code class="language-plaintext highlighter-rouge">ssh</code> on her local machine. The configuration (<code class="language-plaintext highlighter-rouge">~/.ssh/config</code>) is along the lines of the following.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Host server2 HostName server1-domain User alice-username IdentityFile ~/.ssh/id_rsa Host server1 HostName server1-domain User alice-username ForwardX11Trusted <span class="nb">yes </span>ForwardAgent <span class="nb">yes </span>IdentityFile ~/.ssh/id_rsa ProxyCommand ssh server2 <span class="nt">-W</span> %h:%p </code></pre></div></div> <p>Now, everything should be in place. <em>Alice</em> should be able to do login, port forwarding, or whatnot, with automatic key-pair authentication (without having to type her password <strong>every single time</strong> for <strong>every single server along the way</strong>!).</p> <p class="notice--danger">🛑 NB: Data loads between <em>Server1</em> and <em>Server2</em> is not encrypted.</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Log in *Server1* via *Server2*.</span> ssh server1 <span class="c"># Tunnelling ports to *Server1* via *Server2*.</span> ssh <span class="nt">-NL</span> <span class="o">{</span>listening_port<span class="o">}</span>:<span class="o">{</span>hostmachine<span class="o">}</span>:<span class="o">{</span>host_port<span class="o">}</span> server1 </code></pre></div></div> <p>p.s. Wrestling with company proxies during these COVID times we live in at the moment can be devastating 😷</p>Hongyu Hèsystems basicsAutoencoders and VAEs2020-02-15T00:00:00+00:002020-02-15T00:00:00+00:00http://hongyu.nl/autoencoders<h2 id="1-autoencoders">1 Autoencoders</h2> <p><img src="http://hongyu.nl/_resources/ebe53b3973334dbcb66cef074485f5d1.png" alt="60bc1ab0a0006196017f2d3cf12edf4e.png" /></p> <p>The main idea of autoencoders is to extract latent features that are not easily observable yet play an important role in one or several aspects of the data (e.g., images).</p> <p><img src="http://hongyu.nl/_resources/f59db3fa307d465299e34dd02ddd056f.png" alt="010ba1fa8ed15183bffb974199a389c3.png" /></p> <figure class="align-center"> <img src="http://hongyu.nl/_resources/a1aa34b06a484b4f90d9dc2f1010af90.png" alt="" /> <figcaption>Embedding of faces [Saul &amp; Roweis]</figcaption> </figure> <!-- ![fa1b75f261850b8706e4b731ed2fd55d.png](http://hongyu.nl/_resources/a1aa34b06a484b4f90d9dc2f1010af90.png) --> <h3 id="11-compression-by-the-encoder">1.1 Compression (by the “encoder”)</h3> <p><img src="http://hongyu.nl/_resources/b557ab08ab414bceac4a76f45b6af308.png" alt="4943cd912dd88cc5f1f82d85d73ca482.png" /></p> <p>The first step of the process is to compress the observed data vector $\vec x$ into the latent feature vector $\vec z$.</p> <p>There are two obvious benefits yielded from such compression process:</p> <ol> <li>The latent feature vector $\vec z$ is much smaller in size, which makes them much easier to process compared to the original (potentially high dimensional) data.</li> <li>As its name suggests, the latent feature vector $\vec z$ may capture important hidden features. </li> </ol> <h3 id="12-reconstruction-by-the-decoder">1.2 Reconstruction (by the “decoder”)</h3> <p><img src="http://hongyu.nl/_resources/35e7be83efe74e23b98db75467b84c42.png" alt="f47f300caacbfbfd0b3d76705832f118.png" /></p> <p>The second phase is to try to reproduce the data (the image) from the latent feature vector $\vec z$. </p> <p>Apparently, since the first step is a “lossy compression”, the data reconstructed $\vec{\hat{x}}$ will not be exactly the same as the original observation. Here is where the third phase comes about.</p> <h3 id="13-backpropagation">1.3 Backpropagation</h3> <p>As mentioned above, there is a difference between the observation $\vec{x}$ and the reconstruction $\vec{\hat{x}}$.</p> <p><img src="http://hongyu.nl/_resources/58b3571b434f4ac09f0d65ba672836d6.png" alt="00c472c860f7f276f01d91f7ff4bade2.png" /></p> <p>From the above picture, we can see clearly that the higher the dimension of the latent feature vector $\vec{z}$, the higher the quality of the reconstruction.</p> <p>Therefore, constraining the size of the latent space will enforce the “importance” of the extracted features. </p> <p>Further, we can use a loss function to measure such “importance” of the extracted hidden variables. In this case, we use a simple square loss:</p> $\mathcal{L}(x, \hat{x})=\|x-\hat{x}\|^{2}$ <p>Thus, the key power of autoencoders is that</p> <p><strong>Autoencoder allows us to quantify the latent variables without labels (gold-standard data)!</strong></p> <p>To summarize, </p> <ul> <li>Autoencoding == <strong>Auto</strong>matically <strong>encoding</strong> data</li> <li>Bottleneck hidden layer forces the network to learn a compressed latent representation.</li> <li>Reconstruction loss forces the latent representation be as “paramount” and “informative” as possible.</li> </ul> <h2 id="2-variational-autoencoders-vaes">2 Variational Autoencoders (VAEs)</h2> <p><img src="http://hongyu.nl/_resources/739dc342d3714d96a2b1d39e5eb07f7c.png" alt="5bfdc446d533a2f9a737ddea851534d1.png" /></p> <h3 id="21-stochastical-variation">2.1 Stochastical variation</h3> <figure style="width: 30%" class="align-left"> <img src="http://hongyu.nl/_resources/b4cd6671e16e4d0fb188b81a0a31b45c.png" alt="" /> <!-- <figcaption>System overview</figcaption> --> </figure> <p>In a nutshell, variational autoencoders are a probabilistic twist on autoencoders, i.e. (stochastically) sample from the mean and standard deviation to compute the latent sample as supposed to deterministically take the entire latent vector $\vec{z}$. That been said, the main idea of the forward propagation does not change compared to traditional autoencoders. </p> <ul> <li>In the compression process, the encoder computes $p_{\phi}(\mathrm{z} \mid x)$.</li> <li>In the reconstruction phase, the decoder computes $q_{\theta}(\mathrm{x} \mid z)$.</li> </ul> <p>Then, we could compute the loss as follows</p> $\mathcal{L}(\phi, \theta, x)=(\text { reconstruction loss })+(\text { regularization term }),$ <p>which is exactly the same as before. It captures the pixel-wise difference between the input and the reconstructed output. This is a metrics of how well the network is doing at generating the distribution that akin to that of the observation.</p> <p>As to the “regularization term”, since the VAE is producing these probability distributions, we want to place some constraints on how they are computed as well as what that probability distribution resembles as a part of regularizing and training the network.</p> <p>Hence, we place a prior $p(z)$ on the latent distribution as follows</p> $D(p_{\phi}(z|x)\ ||\ p(z)),$ <p>which captures the KL divergence between the inferred latent distribution and this fixed prior for which a common choice is a normal Gaussian, i.e. we centre it around with a mean of 0 and a standard deviation 1: $\ p(z)=\mathcal{N}\left(\mu=0, \sigma^{2}=1\right)$.</p> <p>In this way, the network will learn to penalise itself when it tries to cheat and cluster points outside sort of this smooth Gaussian distribution as it would be the case if it was overfitting or trying to memorize particular instances of the input.</p> <p>Thus, this will enforce the extracted $\vec z$ follows the shape of our initial hypothesis about the distribution, smoothing out the latent space and, in turn, helping the network not over-fit on certain parts of the latent space.</p> <h3 id="22-backpropagation-reparametrization">2.2 Backpropagation? Reparametrization</h3> <figure style="width: 47%" class="align-center"> <img src="http://hongyu.nl/_resources/c44854a3bb34468b9744e8ae8b5011f4.png" alt="" /> <figcaption>Original form</figcaption> </figure> <p>Unfortunately, due to the stochastic nature, the backpropagation cannot pass the sampling layer as backpropagation requires deterministic nodes to be able to iteratively pass gradients and apply the chain rule through.</p> <figure style="width: 53%" class="align-center"> <img src="http://hongyu.nl/_resources/0c92b033a4db47db89be4f4713492eeb.png" alt="" /> <figcaption>Reparametrized form</figcaption> </figure> <!-- ![1512d4ff467d25ef402352d65afa8f15.png](http://hongyu.nl/_resources/0c92b033a4db47db89be4f4713492eeb.png) --> <p>Instead, we consider the sampled latent vector $\vec z$ as a sum of a fixed vector $\vec \mu$ a fixed variance vector $\vec \sigma$ and then scaled this variance vector by a random constant that is drawn from a prior distribution, for example from a normal Gaussian. The key idea here is that we still have a stochastic node but since we have done this reparametrization with the factor $\epsilon$ that is drawn from a normal distribution, this stochastic sampling does not occur directly in the bottleneck layer of $\vec z$. This way, we can reparametrize where that sampling is occurring.</p> <p>Note that this is a really powerful trick as such reparametrization is what allows for VAEs to be trained end-to-end.</p> <h2 id="3-code-example">3 Code example</h2> <p>The following is a vanila implementation of a VAE model in Tensorflow.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Sampling</span><span class="p">(</span><span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="n">Layer</span><span class="p">):</span> <span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">inputs</span><span class="p">)</span> <span class="p">:</span> <span class="n">z_mean</span><span class="p">,</span> <span class="n">z_log_var</span> <span class="o">=</span> <span class="n">inputs</span> <span class="n">batch</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">shape</span><span class="p">(</span><span class="n">z_mean</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="n">dim</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">shape</span><span class="p">(</span><span class="n">z_mean</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span> <span class="n">epsilon</span> <span class="o">=</span> <span class="n">tf</span><span class="p">.</span><span class="n">keras</span><span class="p">.</span><span class="n">backend</span><span class="p">.</span><span class="n">random_normal_</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="n">batch</span><span class="p">,</span> <span class="n">dim</span><span class="p">))</span> <span class="k">return</span> <span class="n">z_mean</span> <span class="o">+</span> <span class="n">tf</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="mf">0.5</span> <span class="o">*</span> <span class="n">z_log_var</span><span class="p">)</span> <span class="n">latenet_dim</span> <span class="o">=</span> <span class="mi">2</span> <span class="n">encoder_inputs</span> <span class="o">=</span> <span class="n">Input</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">6</span><span class="p">),</span> <span class="n">name</span><span class="o">=</span><span class="s">"input_layer"</span><span class="p">)</span> <span class="n">X</span> <span class="o">=</span> <span class="n">Dense</span> <span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">"relu"</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"h1"</span><span class="p">)(</span><span class="n">encoder_inputs</span><span class="p">)</span> <span class="n">X</span> <span class="o">=</span> <span class="n">Dense</span> <span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">"relu"</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"h2"</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="n">X</span> <span class="o">=</span> <span class="n">Dense</span> <span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s">"relu"</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"h3"</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="n">z_mean</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">latent_dim</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"z_mean"</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="n">z_log_var</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">latent_dim</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">"z_log_var"</span><span class="p">)(</span><span class="n">x</span><span class="p">)</span> <span class="n">z</span> <span class="o">=</span> <span class="n">Sampling</span><span class="p">()([</span><span class="n">z_mean</span><span class="p">,</span> <span class="n">z_log_var</span><span class="p">])</span> <span class="n">encoder</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">Model</span><span class="p">(</span><span class="n">encoder_inputs</span><span class="p">,</span> <span class="p">[</span><span class="n">z_mean</span><span class="p">,</span> <span class="n">z_log_var</span><span class="p">,</span> <span class="n">z</span><span class="p">],</span> <span class="n">name</span><span class="o">=</span><span class="s">"encoder"</span><span class="p">)</span> <span class="n">keras</span><span class="p">.</span><span class="n">utils</span><span class="p">.</span><span class="n">plot_model</span><span class="p">(</span><span class="n">encoder</span><span class="p">,</span> <span class="n">show_shapes</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> </code></pre></div></div>Hongyu Hèmachine learningHey **system call**! Where on earth are you?2019-09-05T00:00:00+00:002019-09-05T00:00:00+00:00http://hongyu.nl/syscall<p>We have been talking about system call for a while, e.g. applications request services from the operating system via <code class="language-plaintext highlighter-rouge">syscall</code>, but we have never ever seen a system call!</p> <hr /> <h2 id="1-where-are-they--layering">1. Where are they? — Layering</h2> <p>First, let’s recall the UNIX system structure:</p> <p><img src="http://hongyu.nl/_resources/ee3ccb17abb144658616074238b036cc.png" alt="67aa64408fc389648b9833447e860890.png" /></p> <p>System call is an interface between the user and kernel mode which is not necessarily the interface that you want to give a programmer for security concerns.</p> <p><img src="http://hongyu.nl/_resources/613135d3020d4ea0a467fcab532d178f.png" alt="900058532f678fee1601a7b87729a00a.png" /></p> <p>Consequently, system call is buried in the programming language run time library (e.g. <code class="language-plaintext highlighter-rouge">libc.a</code>) so it is the C library that actually makes the system calls to the operating system for us.</p> <hr /> <h2 id="2-the-charming-narrow-waist--the-power-of-uniformed-api">2. The charming “narrow waist” — The power of uniformed API</h2> <p><img src="http://hongyu.nl/_resources/d5b797aa35704c23a6e2842bcce606b4.png" alt="cf22a471c3f58082684fcdd262f40c37.png" /></p> <p>System calls show their power when we are dealing with multiple different devices. When their <code class="language-plaintext highlighter-rouge">syscall</code> are similar enough, mounting them become easy.</p> <p><em>// It is said that the way Linux deals with it is to encompass every system call under the sun from all kinds of different operating systems. Terrific ~</em></p> <p>Normally, we will get a different chunk of data reading from different devices but, due to the virtue of uniformity, we are able to read from and write to disk driver in exactly the same way as we read from and write to a flash memory. This is because the interface of the kernel is byte-oriented, which means it is reading and writing bytes so it doesn’t care the size of the data blocks.</p> <hr /> <h2 id="3-final-notes-kernel-buffered-read--write">3. Final notes: kernel-buffered <code class="language-plaintext highlighter-rouge">read</code> &amp;&amp; <code class="language-plaintext highlighter-rouge">write</code></h2> <h3 id="recall-the-six-key-design-concepts-of-unix-io">Recall the six key design concepts of UNIX I/O</h3> <ul> <li> <p><strong>Uniformity</strong></p> <ul> <li>File operations, device I/O, and interprocess communication through <code class="language-plaintext highlighter-rouge">open</code>, <code class="language-plaintext highlighter-rouge">read</code>/<code class="language-plaintext highlighter-rouge">write</code> and <code class="language-plaintext highlighter-rouge">close</code>.</li> <li>Allow simple composition of programs, for example, <code class="language-plaintext highlighter-rouge">find | grep | wc …</code>.</li> </ul> </li> <li> <p><strong>Open before use</strong></p> <ul> <li>Provides opportunity for access control and arbitration.</li> <li>Sets up the underlying machinery, i.e., data structures.</li> </ul> </li> <li> <p><strong>Byte-oriented</strong></p> <ul> <li>Even if data blocks are transferred, addressing is in bytes.</li> </ul> </li> <li> <p><strong>Kernel buffered reads</strong></p> <ul> <li>Streaming and block devices look the same.</li> <li>Read blocks processes, yielding processor to other task.</li> </ul> </li> <li> <p><strong>Kernel buffered writes</strong></p> <ul> <li>Completion of out-going transfer decoupled from the application, allowing it to continue.</li> </ul> </li> <li> <p><strong>Explicit close</strong></p> </li> </ul> <p>As we discussed in the training session, reading something off a disk is time-consuming and costly, often up to several milli-seconds, which is roughly equal to a million instruction times. Thus, in order not to lose a million instructions, we better put the corresponding processes into sleep yielding processors to other tasks.</p> <p>Same for writing, when the system call <code class="language-plaintext highlighter-rouge">write</code> returns, the data is not necessarily on the disk but buffered in the memory (the kernel), allowing the applications to keep going.</p> <p>In a nutshell, the kernel is doing tons of buffering and visualization behind the scenes.</p> <p>(In other words, if your machine crashes at a wrong point in time, you will lose your data permanently …)</p> <p><em>p.s. I’ll (hopefully) get to the user-buffered I/O later.</em></p>Hongyu Hèoperating systemfork( ) yourself!2019-09-03T00:00:00+00:002019-09-03T00:00:00+00:00http://hongyu.nl/fork<h2 id="1-recall">1. Recall</h2> <p>As we mentioned before, a process is an instance of a program executing. Its state and information are all monitored by the OS. Processes can do their work by invoking system calls.</p> <p><strong>But</strong> is there any operations that are on their own?</p> <p>Yes, process can create a new process (sometimes called sub-process) by copying itself!</p> <hr /> <h2 id="2-fork">2. <code class="language-plaintext highlighter-rouge">fork()</code></h2> <ul> <li> <p>In typical UNIX systems (exclude Linux in a sense that it may somehow augment the child process in the first place), the <code class="language-plaintext highlighter-rouge">fork()</code> system call (or library precisely) creates a copy of the callee process.</p> </li> <li> <p>When I say <strong>copy</strong>, I mean all of the states of the original process duplicated in both the parent and the child! (Memory, File Descriptors, etc…)</p> </li> </ul> <p><img src="http://hongyu.nl/_resources/fc63e8cf3a2b4ccd99317c83218c2e8b.png" alt="191689222fa2f0ce00ab8e0d856f0e19.png" /></p> <ul> <li><code class="language-plaintext highlighter-rouge">fork()</code> returns 3 kinds of values: <ul> <li> 0  represents that this process is the parent;</li> <li> 1  represents that this is process is the child;</li> <li><code class="language-plaintext highlighter-rouge">-1</code> (as UNIX convention) represents an error message.</li> </ul> </li> <li>Why we need those return values? <ul> <li>when we call <code class="language-plaintext highlighter-rouge">fork()</code>, the original process will be trapped into the kernel mode and halt until it returns.</li> <li>when it returns, the kernel will actually return twice, once in the parent process and once in the child.</li> <li>the parent and the child starts from the instruction after the <code class="language-plaintext highlighter-rouge">fork()</code> together.</li> </ul> </li> </ul> <p>An example:</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include &lt;stdlib.h&gt; #include &lt;stdio.h&gt; #include &lt;string.h&gt; #include &lt;unistd.h&gt; #include &lt;sys/types.h&gt; #define BUFSIZE 1024 </span><span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span> <span class="o">*</span><span class="n">argv</span><span class="p">[])</span> <span class="p">{</span> <span class="kt">char</span> <span class="n">buf</span><span class="p">[</span><span class="n">BUFSIZE</span><span class="p">];</span> <span class="kt">size_t</span> <span class="n">readlen</span><span class="p">,</span> <span class="n">writelen</span><span class="p">,</span> <span class="n">slen</span><span class="p">;</span> <span class="n">pid_t</span> <span class="n">cpid</span><span class="p">,</span> <span class="n">mypid</span><span class="p">;</span> <span class="n">pid_t</span> <span class="n">pid</span> <span class="o">=</span> <span class="n">getpid</span><span class="p">();</span> <span class="cm">/* get current processes PID */</span> <span class="n">printf</span><span class="p">(</span><span class="s">"Parent pid: %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">pid</span><span class="p">);</span> <span class="n">cpid</span> <span class="o">=</span> <span class="n">fork</span><span class="p">();</span> <span class="k">if</span> <span class="p">(</span><span class="n">cpid</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span> <span class="cm">/* Parent Process */</span> <span class="n">mypid</span> <span class="o">=</span> <span class="n">getpid</span><span class="p">();</span> <span class="n">printf</span><span class="p">(</span><span class="s">"[%d] parent of [%d]</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">mypid</span><span class="p">,</span> <span class="n">cpid</span><span class="p">);</span> <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">cpid</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span> <span class="cm">/* Child Process */</span> <span class="n">mypid</span> <span class="o">=</span> <span class="n">getpid</span><span class="p">();</span> <span class="n">printf</span><span class="p">(</span><span class="s">"[%d] child</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">mypid</span><span class="p">);</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="n">perror</span><span class="p">(</span><span class="s">"Fork failed"</span><span class="p">);</span> <span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span> <span class="p">}</span> <span class="n">exit</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span> <span class="p">}</span> </code></pre></div></div> <hr /> <h2 id="3-the-relationship-between-parent-and-child-processes">3. The relationship between parent and child processes</h2> <ul> <li> <p><code class="language-plaintext highlighter-rouge">fork()</code> is executed in an unblocked manner, which means the parent process will not naturally sit there and wait for their child processes to return.</p> </li> <li> <p>Furthermore, they are actually running in parallel that both of them are exchanging time on the scheduler queue and the run queue.</p> </li> <li> <p>The parent processes are able to control their children processes directly.</p> </li> <li>What if the child gets “killed” right away before its parent does anything? <ul> <li>Basically, when this happens, the PCB of the child will not be reallocated by the OS;</li> <li>Instead, these dead children will stay in a state called the <strong>Zombie State</strong>. (When you mess up the so-called <strong>interprocess communication</strong> between parent and children process, there will be a bunch of “Zombie” appear)</li> </ul> </li> <li>What if the parent gets “killed” before the child process return? <ul> <li>Generally, the child will get inherited by a grandparent.</li> </ul> </li> </ul> <p><em>// Didn’t expect it’s going to be such a creepy parenting blog …</em></p> <hr /> <h2 id="4-shell">4. Shell</h2> <blockquote> <p>A shell is a job control system which allows programmers to create and manage a set of programs to do some task —— <a href="https://inst.eecs.berkeley.edu/~cs162/sp15/">Berkeley CS162</a></p> </blockquote> <blockquote> <p>A shell is a command interpreter which makes key process-management system calls that are dealing with the creation and termination of processes. ——　Prof. Andy Tanenbaum</p> </blockquote> <p>Okay~ how come we wind up with Shell anyway?</p> <ul> <li>Every process has a parent and the parent also has its parent and so forth. We can go on and on and on until we hit something called <code class="language-plaintext highlighter-rouge">init</code> in UNIX;</li> </ul> <p><img src="http://hongyu.nl/_resources/784d1c5712124c1ba545c0b02f7e933c.png" alt="ecddae82fa5a8a474579c7beaa017810.png" /> <img src="http://hongyu.nl/_resources/32dbaacc01ac42d19ebeab9ae65990db.png" alt="4a71fe295fce88f7c0226b9185f40222.png" /></p> <ul> <li>The <code class="language-plaintext highlighter-rouge">init</code> is the first process which calls all other children processes and one of which is the shell.</li> </ul> <p><img src="http://hongyu.nl/_resources/972bdaf92c594fe584452a8ba0e6898c.png" alt="64e229116c5d679cd8d9b8380099d613.png" /></p> <ul> <li>The shell, whose job is to create and manipulate processes us, turns out to be a special process that <code class="language-plaintext highlighter-rouge">fork()</code>s itself and immediately calls <code class="language-plaintext highlighter-rouge">exec()</code> to load a new program into its memory address. (This often followed with <code class="language-plaintext highlighter-rouge">wait()</code> that blocks itself until gets the return value from the child, which releases the process from being a zombie.)</li> </ul> <p><img src="http://hongyu.nl/_resources/5e614aa771964b9d945dcd07e228909c.png" alt="1b3f75dcc541df051af709f790eb89c6.png" /></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cc –c sourcefile1.c cc –c sourcefile2.c <span class="nb">ln</span> –o program sourcefile1.o sourcefile2.o ./program </code></pre></div></div> <ul> <li>The way that the Shell works is also how we start up a parallel program.</li> </ul> <p><img src="http://hongyu.nl/_resources/d18ba048fe3a489a90d331c253715973.png" alt="d0dcb4ad06069ef8cb758ff3033c1192.png" /></p> <hr /> <h2 id="5-unix-process-management">5. UNIX Process Management</h2> <p><img src="http://hongyu.nl/_resources/bf7b286b6b044f41b535a08359950354.png" alt="d93f88765dc551fa3e4d24a102944beb.png" /></p> <ul> <li>UNIX <code class="language-plaintext highlighter-rouge">fork</code> – system call to create a copy of the current process, and start it running. <ul> <li>No arguments!</li> </ul> </li> <li> <p>UNIX <code class="language-plaintext highlighter-rouge">exec</code> – system call to change the program being run by the current process (replace the current running process with a brand new process).</p> </li> <li> <p>UNIX <code class="language-plaintext highlighter-rouge">wait</code> – system call to wait for a process to finish.</p> </li> <li>UNIX <code class="language-plaintext highlighter-rouge">signal</code> – system call to send a notification to another process.</li> </ul> <p>( To me, the <code class="language-plaintext highlighter-rouge">signal</code> service is just a sort of user-level interruption that works in any process regardless it’s a parent or child. )</p> <p>An example of using the <code class="language-plaintext highlighter-rouge">signal()</code>:</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include &lt;stdlib.h&gt; #include &lt;stdio.h&gt; #include &lt;sys/types.h&gt; #include &lt;unistd.h&gt; #include &lt;signal.h&gt; </span> <span class="c1">// this is just a way of changing the original SIGINT handler to our self-defined hander</span> <span class="kt">void</span> <span class="nf">signal_callback_handler</span><span class="p">(</span><span class="kt">int</span> <span class="n">signum</span><span class="p">)</span> <span class="p">{</span> <span class="n">printf</span><span class="p">(</span><span class="s">"Caught signal %d - phew!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">signum</span><span class="p">);</span> <span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span> <span class="p">}</span> <span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">signal</span><span class="p">(</span><span class="n">SIGINT</span><span class="p">,</span> <span class="n">signal_callback_handler</span><span class="p">);</span> <span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div>Hongyu Hèoperating system