The ultimate question

The question both puzzled and fascinated me. Think about it: there’s absolutely nothing for billions of years before a person is born, and after we die, there is again void, most likely forever. So what’s the point of some 70 years of fleeting life? To the universe, there’s no difference between our death and the death of the chicken on our plate, the ant we accidentally crushed, or the leaves in the dust. You might argue that we matter to the world we live in, but what’s this thing we call the “world”? Isn’t it just a concept, much like the notion of “money,” “society,” and “country”? So maybe there’s no point in the first place after all.

I’m not the only one arriving at this conclusion. The philosopher Herbert Fingarette, who wrote books discussing life, came to the same realization at the end of his.^[fingarette] He regretted claiming that fearing death is irrational as he went through that very fear during his final days. In the end, he still couldn’t understand this fear or explain the meaning of life, other than “just going on existing, that’s the truth of it is existing and waiting.”^[being-97] So perhaps life is meaningless after all. Be that as it may, that’s precisely why we have to find meaning ourselves. Indeed, a life without it is just as insignificant as the cooked chicken, the smashed ant, and the fallen leaves. But what is meaning, and how do we find it?

Inner quest for the outside life

To find meaning in the outside world, we have to first look inside our hearts: what do we value? What makes us truly happy and brings us real joy? Two factors make answering these questions especially hard: (1) we don’t know ourselves as well as we should, and (2) we often aren’t honest with ourselves, either. More often than not, instead of looking inward, we’re tangled up in external stimulation. Instead of refining our souls, we strive to improve our appearances. Instead of caring for the ones we love and the ones who love us, we’re busy maintaining or building fleeting relationships.

Years ago, someone told me I needed to trust people more, reminding me that “we aren’t alone in this world.” But the sad reality is that this quick flash of life is ultimately a solo journey. Friends come and go, family members pass away, and we inevitably grow older as well. To me, this world is like a vast archipelago. Each of us lives on a single island, and every island has its own theater where we act out our lives. Sure, some islands sit closer together (like the ones belonging to our parents), but they’re all fundamentally isolated. Because of that, only we can truly understand the firsthand stories of our own performances. When the curtain finally falls, we’re the only audience left. Lying on our deathbeds, all we have left are our withered bodies and the memories playing in our theaters.

For me, the value of doing a PhD isn’t about earning a title or fame, making more money after graduation, or even pushing the frontier of scientific discovery. Rather, it’s a precious journey of self-discovery. It gives me the luxury of time and space to look inward: what am I good at, and what am I not? What brings me real joy, and what do I truly value? The process of struggling with difficult problems, getting excited, becoming frustrated, and having to hold on as well as let go, is itself a once-in-a-lifetime opportunity that helps me shape my character.

Haven with pain

Memories are the bookmarks of our experiences, marking the important moments where we fought, struggled, failed, and survived. We determine what they represent and what colors these bookmarks have. As mentioned in the book Counsels and Maxims by Arthur Schopenhauer, these moments fade away easily unless we intentionally make them stand out, like through celebration and journaling. That’s why I celebrated my birthday and rebirthday this year, although the associated childhood memories weren’t terribly positive. I believe that’s going to be true for most of our memories, as struggles and hardships are an integral part of our lives.

“Pain is inevitable but suffering is optional.” — What I Talk About When I Talk About Running, Haruki Murakami

Life is only beautiful as it is painful. I loved the book from Liu Yong (刘墉), The Paradise with Suffering (寻找一个有苦难的天堂), which pointed out that happiness only exists when there’s pain, just like Yin and Yang. I chase sunshine only when there’s a lack of it; I enjoy spending time alone in tranquility, as I’ve gone through enough chaos; I can see the bright side of people, since I’ve been in the dark. It’s the pain and struggles that help us build our character and are often the most memorable. So even if they unavoidably take something away from us in the meantime, they’re worth celebrating. It’s just like Jensen Huang once said, “I wish upon you ample doses of pain and suffering.”

Are we really special?

AI has taught me more about life than I could’ve ever anticipated, giving me a massive dose of humility along the way. Plenty of computer scientists (like Yann LeCun) dismiss current AI as a mere “stochastic parrot”, arguing that it just predicts the next token without any real thinking or consciousness.^{[lecun][stochastic-parrots]} But that begs the question: what’s consciousness, really? When I’m typing these very words, am I exercising “conscious thought,” or am I just generating the next highly probable word given my context? Regardless, I wouldn’t want to call myself a flesh-and-blood parrot made out of a giant bag of neurons.

Even if we assume for a second that AI genuinely lacks human-like thought or consciousness, it’s already surpassed human performance on many tasks. We might argue that there’re still countless things that AI can’t do, esp. those related to the physical world. But isn’t that a bit unfair? If I were trained to read and do mental math all day long, and then someone asked me to clean a room, I might well not know how to do it very well either.

PS: since I started picking up mental math again (like multiplying 2 two-digit numbers in my head within 5s), I’ve noticed how much my mind can feel like a computer. I need a small space to hold intermediate results, something like registers. Working memory can hold only a few chunks at a time.^{[working-memory]} When another calculation demands that space, a number I was trying to keep there slips away. It can feel like a rather shabby machine…

AI’s human-level or even superhuman performance is an ironic but undeniable proof that a simple next-token-prediction mechanism can pull off most of the things that humans are capable of. Maybe our brand of “intelligence” isn’t quite as magical as we thought. Just as we eventually accepted that we aren’t the center of the universe, maybe it’s time we learned to be a bit more humble about our minds.

I believe the core mechanism behind both artificial intelligence and human intelligence is the same: complexity. As Melanie Mitchell describes in her book, our brains are composed of extremely simple and uniform parts, much like other complex systems such as army ant colonies, the internet, and global economies.^[complexity] While we can thoroughly understand the function of each individual piece, their collective behavior becomes a massive mystery simply because of their sheer scale. I’m convinced that the same rule applies to AI. We might know exactly what’s happening inside a single artificial neuron, but once we scale up to millions or trillions of them, the whole system becomes incomprehensible. Ultimately, humans and AI may well derive their “intelligence” from the exact same source.

I really don’t think human uniqueness lies in whatever tasks AI still can’t pull off. Instead, our true uniqueness might just be our inherent “imperfections”: our emotion, irrationality, fear, laziness, vanity, greed, and envy. These messy but palpable traits are probably what actually separate us from the machines. Ultimately, they’re the very things we have to recognize, acknowledge, reconcile, and accept as part of our journey in the search for meaning.

Time flies when you are having fun. It has already been two years since I decided to join Princeton. Now that I am halfway through my second year, I want to look back on how I have been “philosophizing” my PhD journey.

My first year was full of joy and rejection. They sound like opposites, but my experience would not make sense without both.

What I wanted from year one

I started my PhD with two goals: publish a paper in my first year and fully experience American college life.

I set the first goal because I wanted to get my research moving and find my place in the group as quickly as possible. It turned out to be much harder than I expected, but more on that later. The second goal felt uniquely possible at Princeton, even as a PhD student.

In much of Europe, doctoral students have formal contracts with the university. You are technically an employee, and people treat you like one. Things can feel pretty transactional. Universities do not usually do much to organize social life or build community, so many PhD students mostly stay within their research groups.

Princeton feels very different. It puts real effort into building a grad-student community. The graduate school is small, with around 3,000 students in total, even smaller than Stanford’s medical school. We are all part of the Grad College, which runs a steady stream of social and academic events. Most campus events and clubs are open to both undergrads and grads. I have been hiking and skiing with the outdoor club, for example: it is university-affiliated but mostly run by undergrads.

Most importantly, almost everything is free for students. Princeton has a lot of resources, and it treats its students very well. This kind of campus life would be hard to imagine at most European universities, including top-ranked ones.

Year one in numbers

• Number of paper submissions: 8 (4 as first author)
• Number of paper rejections: 6
• Number of grant proposals I helped with: 2
• Number of undergrads mentored: 4
• Number of books read: 9
• Number of courses taken: 6 (required)
• Number of high tables attended: 5
• Number of grad formals: 1
• Number of hikes: 17
• Number of skiing/skating trips: 2
• Number of concerts on campus: 2
• Number of free meals on campus: +∞

A typical day

• 8:00 AM — Play tennis with Minhao at the Princeton Racquet Center
• 10:00 AM — Take the Tiger shuttle to my office
• Morning — Work
• 12:00 PM — Usually a seminar or talk somewhere on campus
• 1:00 PM — Get free food :)
• Afternoon — Work
• 5:00 PM — Take the shuttle back to my apartment on the Meadows campus
• 6:00 PM — Hit the Meadows gym or play squash
• 8:00 PM — Cook dinner or get free food again :)
• Evening — Work

Rejections

Getting a paper out in my first year was much harder than I expected, especially because I was working in an area that was still new to me. I used to think computer networking was basically “systems with different applications.” Oh boy, I was wrong. The field asks different questions and values different things, from empirical performance optimization to formal guarantees, and it took me a while to adjust.

I still feel incredibly lucky to be my advisor’s student. She is one of the smartest and hardest-working people I have ever met, but more importantly, she is a kind mentor. She stays patient when I get stuck and is willing to learn alongside me, even though my work crosses several pretty different areas.

Early on, I was probably not the easiest student to work with. I have strong opinions about research direction and experimental methods. At the time, I did not fully understand how the community I came from differed from the networking community. I also did not know how to turn my ideas into a research paper instead of a technical report crammed with engineering details.

I still remember my first submission. Right before the deadline, my advisor had to rewrite most of it because my writing was so far off track. She walked me through the process and even stayed up with me until 4 a.m. I am deeply grateful for her patience and for how she kept pushing me to think differently.

I also remember getting my first rejection. Two seconds later, I got a message from her: “I’m sorry about the paper.” Something clicked in that moment. She was not my boss; she was my mentor.

“Your ideas are like your children, and you don’t want them to go into the world in rags.” — Patrick Winston

Rejection is hard because our ideas are our children. But eventually I realized the sting was not just “they said no.” It was having to look at my work with fewer illusions. What was the core claim, really? What was the clean story? What would a skeptical reviewer misunderstand, and why might that be my fault? Each rejection pushed me to tighten the logic, cut the noise, and explain the contribution more clearly. In that sense, reviews became less of a verdict and more of a mirror.

So when the paper was finally accepted, I was genuinely happy but not surprised. By then, we had stress-tested the work enough that acceptance felt like the outcome finally catching up with the paper we had fought to make real.

I also learned not to over-read any single decision. Review is noisy, and with LLMs entering the workflow, that noise can feel even louder. A rejection does not automatically mean the work is bad. Maybe the fit was wrong, the framing did not land, or the message got lost. The part you can control is your trajectory: keep learning, keep revising, and keep making the work clearer and stronger.

Still enjoying my solitude

My life has become much more colorful since I joined Princeton. I have made new friends and picked up new hobbies. But one thing has stayed the same: people may worry that I am lonely, while I worry much more about someone getting too far into my life.

I am often alone but never lonely. I love jogging around Lake Carnegie and along the Delaware Canal towpath, where there are plenty of 🦊s and 🦌s. I love walking from my apartment to my office in the morning and walking back alone at night. I even love standing by Lake Carnegie and letting the sun hit my face.

Being alone gives me room to think and decompress. It is hard for some people around me to understand, my mom included, but solitude feels like freedom and control over my own life. At the end of the day, life is a journey you take alone whether you like it or not. If you have not found joy in solitude, I hope you find your own version of it, even in small doses.

“A man can be himself only so long as he is alone; and if he does not love solitude, he will not love freedom; for it is only when he is alone that he is really free.” — Arthur Schopenhauer

PS: I have also learned that, as I get older, the way I treat friends, especially female friends, can be misunderstood. So I have become more deliberate about keeping some space and setting boundaries.

Finding meaning in a PhD (and in life)

To me, doing a PhD somewhere like Princeton is an absolute privilege. It gives me time and space to think about what I really want, what I am good at, and what I am not. For me, a PhD is not mainly about innovation, “moving the needle,” or advancing humanity. It is about understanding: understanding science and, maybe most importantly, understanding life.

Since coming to Princeton, I have started reading physical books again. A good friend introduced me to Gödel, Escher, Bach: an Eternal Golden Braid, and I got hooked. It is about formal logic and its connections to math, art, and thinking. The idea that grabbed me was that mathematical and logical symbols have no inherent meaning. But once we give them meaning, they can express all of mathematics and more.

That got me thinking: why should life be any different? We have moved past the simple search for our next meal or the drive to reproduce. I do not think life comes with one inherent, predetermined meaning. We have to give meaning to the world around us ourselves.

Life is hard. It comes with obstacles and all kinds of internal and external struggles. If we do not give it meaning, what is the point of being here and going through all of it?

A few final thoughts

My first year was full of joy and rejection, and I loved it. I came to understand both my research and myself much better. I am grateful to Princeton for giving me a safe, peaceful place to think. I am equally grateful to my advisor and friends for creating such a supportive, intellectually stimulating environment.

AI is upending the world as we know it, but an ivory tower like Princeton has shielded us from much of that chaos. It gives us time to look inward instead of constantly worrying about where the world is heading. We will all have to face that chaos eventually; for now, I cherish this space.

When I am old and no longer as active, I hope to have a rich memory book full of beautiful moments to flip through, not just a dense CV. Life is not a race to the finish line or a matter of grinding through daily motions. I have come to think it is not really about the process or the destination. It is about understanding who we are and who we become along the way.

Prelude

Around this time two years ago, I was working on a take-home task for a Harvard PhD interview and compulsively refreshing my inbox, spam folder included.

Growing up, I thought China’s National College Entrance Examination (NCEE), or Gaokao, was the most stressful exam a student could face.^[gaokao] It is a high-stakes, years-long filter that shapes college options for millions of students. When I later decided to leave law school, my mom took it hard. That whole period was also rough for me because of a long-running, painful family situation.

Applying to grad school later taught me that stress can look very different. Instead of one clearly defined test, you get uncertainty: it wears you down slowly, with no obvious endpoint. You can do a lot of things “right” and still feel like you are guessing what actually matters.

That is not to downplay the NCEE. It is brutally hard because millions of students compete for a limited number of seats. But it is also transparent: you know what subjects you will be tested on, and once you clear the cutoff, the next step is usually straightforward.

Grad admissions, at least in my experience, can feel like the opposite. It is not always clear what matters most, what only matters sometimes, or what you are over-optimizing without realizing it. If you are applying from outside North America, that information gap can feel even wider.

❗Quick disclaimer: this is just my experience applying to EE/CS PhD programs in engineering-oriented areas such as systems and networking. Things can look very different across fields, departments, labs, and advisors.

What happened

Here is the full list of programs I applied to in the 2023–2024 cycle and how things turned out. The order is arbitrary.

Two quick notes:

• Some schools, including Stanford and Princeton, let you apply to only one program per cycle. Others, including MIT and CMU, allow multiple applications in the same cycle.
• Advising can be flexible across departments, but the admissions rules can differ a lot. At Princeton, ECE students are admitted by the department, rotate during their first year, and are matched with advisors afterward. COS students are admitted straight into advisors’ labs. Rotations are a great way to explore fit, but they can also be stressful.

What actually mattered

I applied from Europe, and the information gap was one of the hardest parts. I did not always know what to prioritize, what was merely nice to have, and what was mostly noise. Most questions I get are about this, so I will start here.

Detailed recommendation letters

In my view, rec letters are the most important part of the application.

Most North American programs ask for 3–5 letters, and the portal will ask whether you want to waive your right to read them. My strong recommendation is to waive access (select “Yes”). Committees often treat waived letters as more candid, and therefore more credible.

What matters even more than “prestige” is specificity. A detailed, sincere letter from a professor who knows your work well can carry more weight than a vague letter from a famous name. Strong letters do more than list achievements. They explain what you contributed, how you think, how you collaborate, and what you are like as a teammate and researcher.

At some schools, including UofT and MIT, an internal committee may screen applications before they reach individual faculty. In those settings, detailed letters can make a real difference because they offer concrete evidence of research readiness, independence, and reliability.

Do grades really matter?

Grades matter, but they are usually not the main signal in PhD admissions.

If you are early in undergrad and considering a PhD, take coursework seriously because it builds your foundations. Just recognize that research experience usually carries the most weight. Many faculty treat grades as supporting evidence and lean on them mostly when they do not have other ways to judge your readiness.

International applicants also run into a practical problem: grading systems are hard to compare. During one University of Toronto interview, a professor told me outright that he was not sure how to evaluate my grades on the Swiss 1–6 scale.

Coursework and research are also just different beasts. Even demanding classes do not automatically build the skills you need for open-ended research. For more on how I think about this, see my mentorship statement.^[mentorship]

Publications and research experience

The point of joining a lab before you apply is to get real research experience, and publications are one of the clearest ways to show it. A top-tier paper can completely change how people read your application.

That said, paper count is not a clean measure of talent. Timing, mentorship, project choice, and luck all matter. Some students get early access to well-scoped problems and strong research pipelines; others do equally good work with fewer structural advantages. Admissions often uses first-author papers as a proxy for research training, so they matter a lot in practice anyway.

This is where I have mixed feelings. I think a PhD should work more like an apprenticeship, and I will come back to that when I explain how I made my final decision.

ℹ️ As Prof. Peter Henderson has argued, the bar for incoming graduate students has climbed too high, and the diversity of students who can clear it has started to collapse …

Faculty often look for similar traits

At admitted-student visits, I kept running into the same people at different schools. It felt like we were all following each other around the country. Over time, it became clear that programs often select for very similar profiles and signals.

That makes patterns worth studying. Read a range of statements of purpose and application materials.^{[sop-collection] [sop-mit]} Look for:

• what kinds of research experiences they had
• what kinds of projects they worked on (and with whom)
• whether they had publications (and at what venues)
• what skills and framing show up repeatedly

For a structured discussion of how PIs and committees evaluate candidates, Chapter 2 of The CS Assistant Professor Handbook^{[asstprofbook]} is great. MIT also has a useful page on what faculty look for in application essays.^{[faculty-hints]}

Before you apply

Beyond the information gap, international applicants may have fewer informal channels for advice, mentorship, and introductions. That makes your timeline and outreach strategy even more important.

Who makes the admissions decision?

You will often hear about two admissions models: “professor-centered” and “committee-based.” In the first, individual professors or principal investigators (PIs) have a lot of influence. In the second, a committee of faculty, and sometimes students, plays a bigger role.

In practice, faculty preferences almost always matter a lot, whatever the formal structure. The main reason is funding: unless a student brings outside funding that covers their PhD (often called a “free student”), their support usually comes from a faculty member’s grants.

To avoid hiring too many or too few students, departments usually ask faculty how many students they expect to fund in the coming cycle. If one or more faculty members genuinely want to work with you and have the money to support you, your odds are very good.

Start early and reach out

I started emailing professors and interviewing in May 2023, roughly seven months before most deadlines. My first interview was with Cornell on May 18, and it turned into one of four conversations I had with Cornell faculty.

Here are three reasons to start early:

1. You learn about programs faster and assess fit sooner.
   To find a good match, you need to understand both the department and the people in it. This mattered especially for me because I was applying from Europe. Some schools combine departments, such as EECS, while others split them into CS/CIS and EE/ECE. If your background sits between hardware and software systems, several program labels may fit, but the culture and research focus can vary wildly.

   Even within “ECE,” departments can look very different. UIUC and Cornell ECE have a strong computing focus, while Princeton ECE spans VLSI, networking, AI, materials, bio, photonics, quantum, and more. If you are interested in optical computing, Princeton ECE might be a good fit; at another school, materials or applied physics may make more sense.

2. You learn who is realistically hiring.
   Often, only a handful of faculty are both a strong fit and actually hiring in a given cycle. I withdrew several applications after learning that relevant faculty were not taking new students. Professors at UW and Columbia told me they had recently over-hired and needed to convert existing thesis students. You will not always learn that from a website, and it matters even more when funding is uncertain.

3. You build interview skills through repetition.
   Many faculty interviews include a short research talk and deep technical questions. Starting early gave me time to improve my slides and, more importantly, get better at explaining my work under pressure.

   I learned some lessons the hard way. My Yale interview was a joint meeting with multiple professors, and I struggled to answer rapid-fire questions before the conversation moved on. My slides also had flaws that experienced people spotted immediately. It did not go great, but it taught me a lot. Afterward, I scheduled one-on-one interviews when I could and improved quickly. I later got offers from every program that interviewed me except Yale, which is fine :P. If you can, use a few early interviews to get reps before the harder ones.

:warning: There is a hidden cost to reaching out early. Those interviews can feel more like “prove it now” than “let’s explore fit.” As Prof. Mae Milano noted, cold outreach may land with faculty who are naturally skeptical and quickly test whether there is a real research match and clear readiness for the work. The questions can be sharper and the conversation less forgiving than a typical in-cycle interview.

The Interviews

Here is how many interviews I had and the kinds of tasks I saw. I ranked programs by how hard the overall process felt to me, but this is one person’s experience, not a universal ranking.

People often ask why I had as many as five interviews at one university. Multiple professors can be interested in the same candidate, and they do not always coordinate closely, so you may end up talking to several of them separately.

The most common tasks I saw were:

• a short research presentation (often with slides)
• technical discussion
• and sometimes reading and reviewing papers

So practice explaining your work clearly, and get comfortable discussing papers with both praise and criticism.

Research presentation and technical discussion

Many interviews start with a short research talk. Fifteen minutes is a common target for a clear overview of your work.

If you have papers, use a few key figures and focus on the core idea, design choices, and evaluation logic. Expect interruptions and questions, especially about trade-offs, alternatives, and “why did you do it this way?”

The best advice I can give is to anticipate questions and practice your answers out loud. Once you learn what faculty tend to probe, many questions repeat across interviews.

You may also talk with senior PhD students or postdocs. I often found those conversations especially fun, and sometimes more technical than the faculty ones. They may also be thinking about day-to-day collaboration, so they are asking whether you would be a good teammate over the long run. If you know who you will meet, read the group’s recent papers and skim relevant code repos beforehand.

Paper review

Here are a few sample reviews I wrote for University of Toronto ^{[reviews-uoft]} and Harvard interviews.^{[reviews-harvard]}

These tasks usually test whether you can:

• accurately identify the paper’s core contributions and assumptions
• offer a thoughtful critique (strengths and limitations)
• propose concrete extensions or alternative approaches

Because the papers often come from the interviewing group, PIs may be looking for a coherent “next step” that builds on their work.

A good place to start is Prof. Onur Mutlu’s paper review guide: mutlu-guide

Coding and other tasks

Some programs throw in other tasks. Several systems faculty at the University of Toronto gave me coding tasks and take-home projects, while the University of Cambridge asked for a 1K-word research proposal.

If you are curious, here is my codebase for a UofT coding interview task on implementing an SCMP ring buffer.^[ringbuffer] It includes both my implementation and experiment results.

Visiting schools

At the time, I lived in Zurich, Switzerland, so flights to the US were long and expensive. Thankfully, every US program on my list except UIUC reimbursed travel and arranged a hotel.

Visits are usually 2–3 days, and scheduling conflicts are common. Cornell’s visit overlapped with Princeton and Penn, so I worked with Cornell to arrange an individual visit.

Caveat: weather can influence your experience

This is not a scientific point, but it is real: weather affects the visit. Boston was cold and rainy when I went, and even though the city is wonderful, it made exploring less fun. Ithaca in late March was snowing like no tomorrow …

Princeton, by contrast, had beautiful weather during my visit, and that plus the campus made a strong impression:

How I chose

People still ask why I chose Princeton, especially after I spent so much time interviewing with other programs and faculty.

It was not an easy decision. I got deep enough into the science of making hard decisions that I made a video^{[decision-talk]} about it. A few friends found it helpful:

Looking back, three things mattered most.

1. I did not love how “job-like” many interviews felt. I understand why selection is necessary when competition is fierce, but parts of the process rubbed against a core belief of mine: a PhD should be an apprenticeship. If applicants are expected to arrive with every skill they need for the PhD, what exactly is the program meant to teach them?

   This stood out because I also had a big-tech job offer at the time. I wanted a PhD precisely because I was looking for a different kind of learning and growth.

   Funny enough, I initially thought my Princeton interview went badly. It turned into an intense research discussion: I never got to present the slides I had prepared and practiced a million times. Instead, I got a stream of probing questions that required real-time reasoning. When it ended, I could not tell how I had done because the PI, now my advisor, kept a neutral expression throughout. Later, I learned that this is just how she looks when she is thinking deeply.

   I think she saw potential in me without putting me through a long list of tests and checks. I am deeply grateful for that trust.

2. The people and the environment mattered a lot.
   During visits, I especially enjoyed meeting my current group members. They were consistently kind, supportive, and thoughtful, and I could picture building a good working life with them. Culture is hard to measure from the outside, but it matters enormously once you are actually doing the PhD.

   After visiting many schools, I also came to feel that Princeton was closest to my mental picture of a classic American campus. After all, the word “campus” originated at Princeton.^{[campus-word]}

3. I wanted to lean into a new research direction.
   Most of my offers were in software or hardware systems, unlike my advisor’s area. I appreciated that she welcomed me even though I did not have a deep background in the particular subfield I would work on during my PhD.

   I have always believed that early in your career, it is worth trying something different before settling into a field. This TED talk makes a similar point: david-talk

ℹ️ Money should not be the only thing you consider, but it matters a lot in practice. If you are constantly worried about making ends meet, it is hard to focus on research. Check stipend levels alongside local cost of living before you decide. Here is a ranking of CS/EE PhD stipends in the US.^{[phd-stipends]}

Looking back

Somehow, it has already been over a year since I started my PhD at Princeton.

Did I make the right choice? I do not think there is one “right” choice in the abstract. You make it right through what you build from it. All I can say is that I have enjoyed every day of my PhD so far. I am grateful for my advisor, the research, and the people around me.

Is it still worth pursuing a CS/EE PhD in the age of AI? That deserves its own post. But if you want a PhD for money or status, do not do it. The PhD has to be for you. And if you do choose it, doing one somewhere like Princeton is an absolute privilege and, in my experience, a real joy ~

Acknowledgments

Thanks to Prof. Mae Milano, Seungju Lee, Constantine Doumanidis, Mathew Madain, Peilin Xin (CMU), Khánh Vũ, and Fengshi Zheng for their helpful comments on an earlier draft. I am also grateful to Prof. Keith Winstein for sharing his advice as a reference.^{[winstein-advice]}

Resources

• Advice posted by Prof. Keith Winstein under the “Writing” section.^{[winstein-advice]}
• The CS Assistant Professor Handbook.^{[asstprofbook]}
• A collection of CS PhD statements of purpose.^{[sop-collection]}
• MIT CommLab guide on writing statements of purpose.^[sop-mit]
• Paper review guide by Prof. Onur Mutlu.^{[mutlu-guide]}
• CS/EE PhD stipend rankings (US).^{[phd-stipends]}
• MIT EECS page on what faculty look for in application essays.^{[faculty-hints]}
• My talk on making hard decisions.^{[decision-talk]}
• My UofT interview paper review examples.^{[reviews-uoft]}
• My UofT take-home coding project.^[ringbuffer]
• My Harvard interview paper review examples.^{[reviews-harvard]}

What Is Queuing Theory, and Why Should You Care?

In computer systems, latency is usually waiting, not doing. Requests wait for CPU time, for a DB connection, for a disk, for a lock, for a thread pool, for a downstream dependency, etc. Queuing theory is the toolkit for reasoning about that waiting:

• When does latency “blow up” as load increases?
• Which component is the bottleneck?
• How many workers/threads/servers do we need to hit an SLO?
• Why do averages look fine while p99/p999 gets ugly?

A useful mental model from systems performance practice is the “utilization knee”: as utilization approaches 100%, queues grow superlinearly and tail latency spikes.

The Basics: How Queues Map to Computer Systems

A queuing system has three parts:

1. Customers: requests, jobs, packets, RPCs, tasks.
2. Servers: CPU cores, threads, DB connections, GPU slots, NICs, disks.
3. Queue/buffer: ready queue, thread pool backlog, socket accept queue, message broker topic, request backlog at a load balancer.

A minimal “single service station” view of many components:

arrivals (λ)  --->  [ queue ]  --->  server(s) with rate μ  --->  departures

Key performance metrics you typically care about:

• Utilization: how busy the servers are.
• Queue length: how much work is waiting.
• Waiting time: how long work sits before service starts.
• Response time (sojourn time): waiting + service.
• Throughput: completed work per unit time.

Kendall’s Notation: A Handy Shorthand

Kendall’s notation is a concise way to name common models: A/S/C.

Examples:

• M/M/1: Poisson arrivals, exponential service, 1 server.
• M/M/c: same, but $c$ parallel servers (also called Erlang-C). ^[mmc-queue]
• G/G/1: “anything goes” arrivals and service times. ^[gg1-queue]

Queue discipline (FIFO, priority, shortest-job-first, processor-sharing) also matters for tail latency and fairness. ^{[queueing-theory]}

Essential Formulas (With Toy Computer-Systems Examples)

1. Little’s Law (works shockingly often)

• $L$: average number of requests in the system (waiting + in service)
• $\lambda$: effective arrival rate (throughput in steady state)
• $W$: average time in system (response time)

Little’s Law is extremely general: it does not require Poisson arrivals or exponential service—just a stable, long-run average system. ^{[littles-law]}

Example: “How many requests are in-flight?”

You run a service doing $1000$ requests/sec steady-state. Average end-to-end latency is $50$ ms.

• $\lambda = 1000$ req/s
• $W = 0.050$ s
• $L = \lambda W = 1000 \times 0.050 = 50$

So, on average, ~50 requests are concurrently in-flight across your service (queued + running). That’s often directly actionable:

• If your service uses one DB connection per in-flight request, you will need O(50) connections (plus headroom).
• If your service has a max in-flight limit of 30, you will either throttle to ~600 req/s or push queuing upstream.

2. Utilization and the stability condition

For $c$ identical servers with service rate $\mu$ each: \(\rho = \frac{\lambda}{c\mu}\)

Interpretation: $\rho$ is the fraction of time the server capacity is busy (on average).

Critical insight: if $\rho \ge 1$, the queue is unstable (it grows without bound). In real systems, “without bound” manifests as timeouts, retries, OOMs, thread exhaustion, and cascading failure.

Example: single-threaded handler

A single-threaded worker processes requests with mean service time 10 ms.

• $\mu = 1/0.010 = 100$ req/s
• If arrivals are $\lambda = 80$ req/s, then $\rho = 0.8$ (stable)
• If arrivals rise to $\lambda = 110$ req/s, then $\rho = 1.1$ (unstable)

This is why “CPU at 100%” is often a crisis: you have no slack to absorb randomness (bursts, long requests, GC pauses).

3. M/M/1: the simplest queue with a sharp lesson

For M/M/1, mean response time is: \(W = \frac{1}{\mu - \lambda}\) and mean waiting time in queue: \(W_q = W - \frac{1}{\mu}\)

Example: an API server with one worker thread

Same service time: 10 ms $\Rightarrow \mu = 100$ req/s.

If $\lambda = 80$ req/s:

• $W = 1/(100-80) = 1/20 = 0.05$ s = 50 ms
• Service time is 10 ms, so
• $W_q = 50 - 10 = 40 ms$ waiting
• By Little’s Law, $L = \lambda W = 80 \times 0.05 = 4$ requests in system, on average

Operational takeaway: Even at 80% utilization, average response time is 5× the service time (because most time is waiting). As $\rho \rightarrow 1$, $W$ diverges rapidly—this is the “latency knee.”

4. M/M/c (Erlang-C): adding workers reduces waiting—sometimes dramatically

For an M/M/c queue, the probability an arrival must wait (all servers busy) is given by the Erlang-C formula (often written in terms of offered load $a=\lambda/\mu$ and utilization $\rho=\lambda/(c\mu)$). ^[mmc-queue]

Rather than stare at the algebra, use it to answer a capacity question.

Example: sizing a thread pool / worker pool

Suppose:

• Arrival rate: $\lambda = 200$ req/s
• Each worker averages 12.5 ms per request $\Rightarrow \mu = 80$ req/s per worker

Case A: 3 workers

• $\rho = 200/(3 \times 80) = 0.833$

Erlang-C results (for these numbers):

• $P(\text{wait}) \approx 0.70$
• Mean queueing delay $W_q \approx 17.6$ ms
• Mean response time $W \approx 30.1$ ms

Case B: 4 workers

• $\rho = 200/(4 \times 80) = 0.625$

Now:

• $P(\text{wait}) \approx 0.32$
• $W_q \approx 2.7$ ms
• $W \approx 15.2$ ms

Interpretation: Adding one worker (3 → 4) did not increase capacity by 33% just to get 33% more throughput. It cut mean latency roughly in half by pulling utilization away from the knee and collapsing the queue.

This is a recurring theme in real services: once you are near saturation, small capacity changes can create huge latency changes.

5. Variability matters (a lot): the VUT / Kingman approximation

Real workloads are rarely “memoryless exponential everything.” Variability in arrivals (bursts) and service (slow queries, GC, cache misses) is what inflates tail latency.

A common, practical approximation for G/G/1 is Kingman’s formula (often called the VUT equation): ^{[kingman-formula]} \(\mathbb{E}[W_q] \approx \left(\frac{\rho}{1-\rho}\right)\left(\frac{c_a^2 + c_s^2}{2}\right)\tau\)

• $\rho$: utilization
• $c_a$: coefficient of variation of inter-arrival times
• $c_s$: coefficient of variation of service times
• $\tau$: mean service time

Example: same mean service time, different variability

Let:

• $\rho = 0.8$
• $\tau = 10$ ms
• arrivals roughly Poisson $\Rightarrow c_a \approx 1$

Scenario 1: highly variable service ($c_s = 1$)

• $\mathbb{E}[W_q] \approx (0.8/0.2) \times ((1^2+1^2)/2) \times 10 \text{ ms}$
• $= 4 \times 1 \times 10 = 40$ ms

Scenario 2: more predictable service ($c_s = 0.2$)

• $\mathbb{E}[W_q] \approx 4 \times ((1 + 0.04)/2) \times 10$
• $= 4 \times 0.52 \times 10 = 20.8$ ms

Same mean utilization and mean service time, but halving variability roughly halves queueing delay.

Operational takeaway: performance work that reduces variance (caching, eliminating stop-the-world pauses, bounding query time, isolating noisy neighbors) often improves tail latency more than shaving a millisecond off the mean. ^{[kingman-formula]}

Queues in Real Computer Systems: Patterns You’ll Actually See

1) A queue at every boundary

Common “queue boundaries” include:

• Load balancer backlog (requests waiting to be accepted)
• Thread pool queue (waiting for a worker)
• DB connection pool (waiting for a connection)
• Downstream service (RPCs queued behind other callers)
• Disk / network (I/O queues)

A key practice: apply Little’s Law per boundary to estimate concurrency needs and identify bottlenecks. ^{[littles-law]}

2) Bottlenecks dominate end-to-end latency

If a request must traverse multiple queues (a small queueing network), the slowest / most utilized stage tends to dominate waiting. Queueing networks are a standard lens for multi-tier services. ^{[queueing-theory]}

3) Queue discipline changes p99 behavior

FIFO is fair-ish, but it lets one slow job delay many fast ones. Alternative disciplines:

• Priority: protect latency-sensitive traffic (but risk starving background jobs). ^{[queueing-theory]}
• Shortest-job-first / SRPT: improves mean latency; may be unfair without safeguards. ^{[queueing-theory]}
• Processor sharing: approximates time-slicing, common in some network and CPU models. ^{[queueing-theory]}

This is why “just add a queue” is not neutral; it encodes policy.

Practical Modeling Workflow (Minimal but Useful)

1. Pick the boundary you’re modeling (e.g., DB connection pool).

2. Measure:

   • $\lambda$: arrival/throughput into that boundary
   • $\tau$: mean service time
   • variability proxies: p50/p95/p99 service times; burstiness of arrivals
3. Compute utilization $\rho$ and check headroom.

4. Use a simple model first:

   • M/M/1 or M/M/c for “first-order” capacity decisions. ^[mmc-queue]
   • Kingman (G/G/1) when variability is clearly the story. ^{[kingman-formula]}
5. Validate against production (does predicted $L$ match observed in-flight? does predicted knee match when p99 explodes?).

The goal is not perfect fidelity; it is a defensible mental model that prevents naive mistakes (like running everything at 95–99% utilization and being surprised by timeouts).

Limitations (What the Formulas Don’t Tell You)

• Non-stationarity: diurnal traffic, deployments, incident conditions.
• Correlated arrivals: synchronized retries, fan-out bursts, cache stampedes.
• Heavy-tailed service times: long-tail queries or rare slow paths can dominate p99/p999.
• Finite buffers and backpressure: real queues drop, throttle, or shed load.
• Retries change $\lambda$: a timeout often creates more work, not less.

Queueing theory still helps here, but you often combine it with measurement, tracing, and load testing.

Wrapping It All Up

If you remember only a few points:

• Little’s Law turns latency and throughput into concurrency: $L=\lambda W$. ^{[littles-law]}
• Utilization is destiny: near 100%, queues (and tail latency) explode.
• Variability inflates waiting: controlling variance can be more valuable than improving the mean. ^{[kingman-formula]}
• Adding capacity near the knee can cut latency disproportionately (Erlang-C intuition). ^[mmc-queue]

References

• Little’s Law (background). ^{[littles-law]}
• System performance and queueing intuition (talk). ^[srecon24]
• M/M/c (Erlang-C) model (background). ^[mmc-queue]
• Queueing theory in practice for performance modeling (talk). ^[lisa17]
• Kingman’s approximation / VUT equation (background). ^{[kingman-formula]}
• Harchol-Balter, M. (2013). Performance Modeling and Design of Computer Systems: Queueing Theory in Action. Cambridge University Press.
• Gross, D., & Harris, C. M. (1998). Fundamentals of Queuing Theory. John Wiley & Sons, Inc.
• Kleinrock, L. (1975). Queueing Systems, Volume 1: Theory. John Wiley & Sons, Inc.
• Sutton, C., & Jordan, M. I. (2010). Bayesian inference for queueing networks and modeling of internet services. ^{[sutton-jordan2010]}

In Linux, both processes and threads are assigned numeric identifiers, and you can see them show up as peer directories under the /proc pseudo-filesystem. Each schedulable entity appears as a subdirectory in the form /proc/[pid], where that number is often referred to as a “PID”.

Here is the catch: the value that shows up in /proc/[pid] is not strictly a process identifier. Depending on context, it may refer to a thread or a process, because Linux historically built threads on top of processes.

You can observe this terminology overload in tools like htop. By default, htop lists both processes and threads without clearly separating them, so its “PID” column contains identifiers that can correspond to either.

This has a historical reason. Early Linux did not have a first-class notion of threads, only processes. Over time, Linux introduced “thread groups” (Linux 2.4, around 2001), which support the POSIX threads model: multiple threads that conceptually belong to one process. Internally, the shared “process ID” that user space expects is implemented as a thread group identifier (TGID). As described in the clone(2) manual, getpid(2) returns the TGID of the caller, not a per-thread identifier.

As a result, threads in the same process share the same TGID, while each thread also has its own unique thread ID (TID). Practically, getpid() returns the same value across all threads in the process, while gettid() returns a unique value per thread.

How to Distinguish Between a Thread and a “Real Process”

Conceptually, Linux processes and threads are similar because the kernel schedules both as runnable entities. The major difference is what they share: threads in the same process typically share an address space and other resources, while separate processes usually do not (unless explicitly arranged).

To distinguish “threads within a process” from a standalone process, the most reliable place to look is:

• /proc/[pid]/task/[tid]

The task/ directory enumerates kernel-visible threads. The tid component is the kernel thread ID. This is distinct from user-level threading abstractions (for example, some managed runtimes can implement user-level “threads” that are not one-to-one kernel threads). Those user-level threads are not directly visible as separate kernel thread IDs.

Within a multithreaded process, all threads belong to the same thread group. The main thread has tid == tgid, and the other threads have distinct tids but the same tgid. You will also notice that /proc/[pid]/task/[tid] mirrors /proc/[pid]/ when pid == tid, because that path is effectively describing the same main thread and the same thread group leader.

So, when you inspect /proc/[pid] for a multithreaded process, you can interpret it as follows:

• The directory name /proc/[pid] corresponds to the process’s TGID (and the main thread’s TID).
• The /proc/[pid]/task/ subdirectories enumerate all kernel threads in that thread group.
• The “process” you intuitively think of is the thread group leader (the main thread) that created the other threads.

Last Note: Multiprocessing vs Multithreading

To avoid confusion with thread groups (TGID), it helps to also remember process groups, which use PGID. TGID is about threads within a process, while PGID is about groups of processes (used heavily for job control in shells).

In broad strokes, a new process is created via fork() (and friends), while a new thread is created via pthread_create() in C. Under the hood, Linux commonly uses the clone() syscall for both, with different flags controlling what is shared.

When a process spawns subprocesses, the spawning process is the parent, and it may either create a new process group or inherit an existing one. If it creates a new process group, the PGID is typically set to the PID of the process that created the group. If it inherits a group, the PGID will match the inherited group.

In contrast, the main thread and the threads it creates are best thought of as siblings under the same thread group: they share a TGID, and they also share the same PGID as the main thread. One practical implication is that threads are not visible as child “processes” to the parent of the thread group leader.

Example

A convenient way to generate a multithreaded workload on Linux is to use stress-ng. In stress-ng terminology, a “stressor” (or “hog”) is a process.

For example, the following command runs a memory contention stressor:

hy@node-0:~$ stress-ng --mcontend 1 -t 10h
stress-ng: info:  [56472] dispatching hogs: 1 mcontend

If you open htop, you can view the resulting process and its threads as a hierarchy. In that display, PGRP corresponds to the process group ID (PGID), while the PID column is overloaded and may represent either process IDs or thread IDs depending on the row.

In the example shown:

• 56472 is the single-threaded parent process created by your shell when you ran the command.
• 56473 is the multithreaded child process (and also the main thread of that child).
• 56474 through 56477 are sibling threads created by the main thread 56473.

If you run pidof on the stressor, you will typically get the TGID (which matches the main thread’s ID):

hy@node-0:~$ pidof stress-ng-mcontend
56473

You can confirm that 56472 is the parent of 56473 by inspecting the parent’s children file:

hy@node-0:~$ cat /proc/56472/task/56472/children
56473

Next, if you list the child’s task/ directory, you will see all kernel threads in that thread group:

hy@node-0:~$ ll /proc/56473/task/
total 0
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 ./
dr-xr-xr-x 9 hy hy 0 Dec 31 22:21 ../
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56473/
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56474/
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56475/
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56476/
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56477/

If you inspect the task/ directory for one of the sibling threads, you will still see the full set of thread IDs in the group (because you are still within the same thread group context):

hy@node-0:~$ ll /proc/56476/task/
total 0
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 ./
dr-xr-xr-x 9 hy hy 0 Dec 31 22:21 ../
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56473/
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56474/
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56475/
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56476/
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56477/

Finally, notice that the single-threaded parent process has only itself under task/:

hy@node-0:~$ ll /proc/56472/task/
total 0
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 ./
dr-xr-xr-x 9 hy hy 0 Dec 31 22:21 ../
dr-xr-xr-x 7 hy hy 0 Dec 31 22:21 56472/

Resource Accounting

Once you start looking at threads, resource accounting becomes another place where Linux tooling can be surprising. Different tools choose to aggregate or split resource usage in different ways, and sometimes they report thread-level information only when explicitly asked.

For example, htop typically aggregates the CPU and memory usage of all threads into the main thread’s row by default. Similarly, ps will usually show aggregated usage for the thread group leader when you query it by PID:

hy@node-0:~$ ps -p 56473 -o %cpu,%mem,cmd
%CPU %MEM CMD
 473  0.0 stress-ng-mcontend

If you try the same query on a sibling thread ID, you may get nothing because ps is often oriented around process identifiers unless you request thread detail:

hy@node-0:~$ ps -p 56476 -o %cpu,%mem,cmd
%CPU %MEM CMD

To display threads, you can use ps -L with the main thread’s ID:

hy@node-0:~$ ps -L 56473 -o %cpu,%mem,cmd
%CPU %MEM CMD
97.3  0.0 stress-ng-mcontend
94.0  0.0 stress-ng-mcontend
94.0  0.0 stress-ng-mcontend
94.0  0.0 stress-ng-mcontend
94.0  0.0 stress-ng-mcontend

If you want a more detailed listing, ps -L ... -F includes fields such as LWP (the thread ID) and NLWP (the number of threads):

hy@node-0:~$ ps -L 56473 -F
UID          PID    PPID     LWP  C NLWP    SZ   RSS PSR STIME TTY      STAT   TIME CMD
hy         56473   56472   56473 97    5 22792  2604  13 08:10 pts/2    RLl+ 302:30 stress-ng-mcontend
hy         56473   56472   56474 94    5 22792  2604   7 08:10 pts/2    RLl+ 292:03 stress-ng-mcontend
hy         56473   56472   56475 94    5 22792  2604  31 08:10 pts/2    RLl+ 292:00 stress-ng-mcontend
hy         56473   56472   56476 94    5 22792  2604  15 08:10 pts/2    RLl+ 291:59 stress-ng-mcontend
hy         56473   56472   56477 94    5 22792  2604   0 08:10 pts/2    RLl+ 292:05 stress-ng-mcontend

Thread reporting in top has similar behavior. With -H, top shows per-thread CPU usage, while without it, top aggregates:

hy@node-0:/proc$ top -H -p 56476
....
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  56473 hy        20   0   91168   2708   2272 R  97.3   0.0 127:24.91 stress-ng-mcont
  56474 hy        20   0   91168   2708   2272 R  94.0   0.0 122:55.16 stress-ng-mcont
  56475 hy        20   0   91168   2708   2272 R  94.0   0.0 122:54.44 stress-ng-mcont
  56476 hy        20   0   91168   2708   2272 R  93.7   0.0 122:55.33 stress-ng-mcont
  56477 hy        20   0   91168   2708   2272 R  92.3   0.0 122:56.57 stress-ng-mcont

Without -H, top reports a single aggregated number and attributes it to the thread group leader:

hy@node-0:~$ top -p 56476
....
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  56473 hy        20   0   91168   2708   2272 R 476.3   0.0 621:39.36 stress-ng-mcont

If you are scripting, you can use batch mode to extract either aggregate or per-thread CPU values, depending on whether you include -H:

# Aggregated
hy@node-0:~$ top -b -n 2 -d 0.2 -p 56476 | tail -1 | awk '{print $9}'
465.0

# Per-thread with -H
hy@node-0:~$ top -b -H -n 2 -d 0.2 -p 56476 | tail -1 | awk '{print $9}'
75.0

If you need ground-truth per-thread accounting, /proc is again the most explicit source. A thread’s /proc/[pid]/stat can reflect aggregated values when accessed via certain paths, whereas /proc/[pid]/task/[tid]/stat is the per-thread view:

# Total CPU time (user and kernel) aggregated across the thread group.
hy@node-0:~$ cat /proc/56476/stat | awk '{print $14, $15}'
9460932 12361

# CPU time for only thread 56476.
hy@node-0:~$ cat /proc/56476/task/56476/stat | awk '{print $14, $15}'
1879429 3032

You can also use psutil for a convenient scripting interface, but it is important to know what it aggregates. In many cases, psutil reports CPU and memory usage in a way that effectively attributes thread-group totals to whichever thread you query, because it is fundamentally process-centric:

>>> import psutil

# Great-grandparent process (e.g., tmux session).
>>> tmux_session = psutil.Process(54711)
>>> tmux_session.ppid()
1
>>> [(child.name(), child.pid) for child in tmux_session.children(recursive=True)]
[('bash', 54712), ('bash', 56236), ('python', 56613), ('stress-ng', 56472), ('stress-ng-mcontend', 56473)]

# Parent process.
>>> parent = psutil.Process(56472)
>>> parent.ppid()
54712
>>> parent.children(recursive=True)
[psutil.Process(pid=56473, name='stress-ng-mcontend', status='running', started='11:21:57')]
>>> parent.num_threads()
1

# Child process (main thread / thread group leader).
>>> child = psutil.Process(56473)
>>> child.num_threads()
5
>>> [thread.id for thread in child.threads()]
[56473, 56474, 56475, 56476, 56477]

# Sibling thread example.
>>> sibling = psutil.Process(56476)
>>> child.ppid()
56472
>>> sibling.ppid()
56472

# Accounting.
>>> parent.cpu_percent(interval=1)
0.0
>>> sibling.cpu_percent(interval=1)
471.5
>>> child.cpu_percent(interval=1)
472.4

>>> tmux_session.cpu_times()
pcputimes(user=7.46, system=3.19, children_user=102.18, children_system=153.15, iowait=0.0)
>>> parent.cpu_times()
pcputimes(user=0.0, system=0.0, children_user=0.0, children_system=0.0, iowait=0.0)
>>> child.cpu_times()
pcputimes(user=45250.11, system=57.79, children_user=0.0, children_system=0.0, iowait=0.0)
>>> sibling.cpu_times()
pcputimes(user=45255.42, system=57.79, children_user=0.0, children_system=0.0, iowait=0.0)

>>> parent.memory_full_info()
pfullmem(rss=6475776, vms=59777024, shared=6078464, text=1728512, lib=0, data=32018432, dirty=0, uss=3051520, pss=3749888, swap=0)
>>> child.memory_full_info()
pfullmem(rss=2772992, vms=93356032, shared=2326528, text=1728512, lib=0, data=65581056, dirty=0, uss=126976, pss=735232, swap=0)
>>> sibling.memory_full_info()
pfullmem(rss=2772992, vms=93356032, shared=2326528, text=1728512, lib=0, data=65581056, dirty=0, uss=126976, pss=735232, swap=0)

>>> tmux_session.memory_percent()
0.007239506814671662
>>> parent.memory_percent()
0.009602063988251591
>>> child.memory_percent()
0.004111699759675096
>>> sibling.memory_percent()
0.004111699759675096

The upshot is that “what is the PID” and “what resources belong to a thread” depend strongly on which abstraction and which tool you are using. Many tools default to aggregating at the thread-group level, even when they print thread IDs, and you typically need explicit flags (or direct /proc inspection) to get consistent per-thread views.

To end the running example, sending an interrupt to one thread can terminate the entire stressor, depending on how the program handles signals:

# Note: this returns True even if the process is a zombie.
>>> parent.is_running() == child.is_running() == sibling.is_running() == True
True

>>> import signal
>>> sibling.send_signal(signal.SIGINT)

>>> parent.is_running() == child.is_running() == sibling.is_running() == False
True

(It seems that interrupting one thread has a bottom-up cascading effect in stress-ng 🥴 )

Happy New Year 🎆 ~

Reference

• https://unix.stackexchange.com/questions/364660/are-threads-implemented-as-processes-on-linux
• https://unix.stackexchange.com/questions/670836/why-do-threads-have-their-own-pid
• https://stackoverflow.com/questions/1221555/retrieve-cpu-usage-and-memory-usage-of-a-single-process-on-linux
• https://unix.stackexchange.com/questions/404054/how-is-a-process-group-id-set
• https://stackoverflow.com/questions/4856255/the-difference-between-fork-vfork-exec-and-clone
• https://stackoverflow.com/questions/19678954/relation-between-thread-id-and-process-id
• https://stackoverflow.com/questions/9430491/find-cpu-usage-for-a-thread-in-linux
• https://stackoverflow.com/questions/1420426/how-to-calculate-the-cpu-usage-of-a-process-by-pid-in-linux-from-c
• https://stackoverflow.com/questions/19919881/sysconf-sc-clk-tck-what-does-it-return
• https://www.baeldung.com/linux/total-process-cpu-usage
• https://psutil.readthedocs.io/en/latest/#processes
• https://man7.org/linux/man-pages/man5/proc.5.html
• https://man7.org/linux/man-pages/man2/getpid.2.html
• https://man7.org/linux/man-pages/man2/gettid.2.html
• https://manpages.ubuntu.com/manpages/bionic/man1/stress-ng.1.html
• https://www.akkadia.org/drepper/nptl-design.pdf

• time until a server fails
• time until a customer churns
• time until an incident is resolved
• time until a job completes
• time until a cache entry expires (or until a key becomes “cold”)

What makes survival analysis special is that it handles incomplete observation windows correctly. In real engineering datasets, you frequently do not observe the event for everyone before you stop collecting data:

• A customer hasn’t churned yet when you export the dataset.
• A disk hasn’t failed yet when you stop the experiment.
• A request hasn’t completed yet when you cut off a trace.

Treating those as “missing” or dropping them biases you toward shorter times and overconfident conclusions. Survival analysis is designed to keep that partial “last-seen” information and remain statistically consistent.

Survival data in systems: what you actually record

A time-to-event dataset typically has at least two columns:

• duration: how long you observed the unit (user/machine/request)
• event: whether the event occurred within your observation window (1) or not (0)

A concrete toy example: time-to-churn (days) for 6 trial users.

Key idea (from the reference video): the “event=0” rows are not “unknown.” They tell you the unit lasted at least that long. ^{[survival-video]}

You also need to be explicit about your time origin:

• For churn: trial start, first purchase, first session, last renewal?
• For failures: installation time, last maintenance, last reboot?
• For incidents: page time, incident creation time, first alert time?

Changing the origin changes the interpretation of the curves and coefficients.

Two core functions: survival and hazard (with intuition)

Survival function answers: “What fraction lasts beyond time t?”

For churn, you can read $S(30)=0.8$ as “80% of users are still active after 30 days.” ^{[survival-analysis]}

Hazard function answers a different question: “Given you’ve made it to time t, how ‘risky’ is right now?”

Formally:

In systems terms, hazard is often closer to what you want operationally:

• “Given a server has been up 20 days, what is its instantaneous failure rate now?”
• “Given a user is still active at week 4, what is the instantaneous churn pressure now?”

A practical relationship to remember:

• high hazard around time t means the survival curve drops steeply around time t
• if hazard increases with time, you’re in “wear-out” behavior (common in hardware); if it decreases, you may be seeing “early-life failures” (infant mortality) or onboarding drop-off

A common derived quantity is the cumulative hazard:

This becomes useful when you want to add hazards over stages or interpret models on a log scale. ^{[survival-analysis]}

The Kaplan–Meier curve: a survival curve from observed events plus “last-seen” records

If you do not want to assume a distribution, the Kaplan–Meier (KM) estimator gives a non-parametric estimate of $S(t)$ using (a) observed event times and (b) “last-seen active” times for units that did not experience the event within the window. ^{[kaplan-meier]}

It is:

Where at each event time $t_i$:

• $n_i$ = number “at risk” just before $t_i$ (still being observed, event not yet happened)
• $d_i$ = number of events at $t_i$

Toy KM computation (churn example).
Using the 6-user table above, sort by time and update the risk set:

Event times are 2, 3, 4, 6 (and we have “last-seen” times at 7 and 10).

• At day 2: $n=6$, $d=1$ → multiply by $(1-1/6)=5/6$
• At day 3: $n=5$, $d=1$ → multiply by $(1-1/5)=4/5$
• At day 4: $n=4$, $d=1$ → multiply by $(1-1/4)=3/4$
• At day 6: $n=3$, $d=1$ → multiply by $(1-1/3)=2/3$

So:

• $\hat S(2)=5/6 \approx 0.833$
• $\hat S(3)=(5/6)(4/5)=4/6 \approx 0.667$
• $\hat S(4)=(5/6)(4/5)(3/4)=3/6 = 0.5$
• $\hat S(6)=(5/6)(4/5)(3/4)(2/3)=2/6 \approx 0.333$

Where “last-seen” matters: users B and D remain “at risk” up to days 7 and 10, respectively, contributing correctly to $n_i$ up to those times. KM is effectively saying: “they didn’t churn before 7/10, and that information counts.” ^{[kaplan-meier]}

A highly practical workflow for product/systems:

• Plot KM curves for cohorts (e.g., different regions, hardware batches, onboarding variants).
• Look for when curves separate: early vs late differences are operationally distinct (onboarding vs retention; infant mortality vs wear-out).

Comparing groups: the log-rank test (what it is and when it lies)

To test whether two groups have different survival curves, the log-rank test compares observed vs expected events over time under the null that the groups share the same survival distribution. ^{[logrank-test]}

The intuition:

• At each event time, if Group A has 60% of the risk set, then under the null it “should” get ~60% of the events.
• The log-rank statistic aggregates how much reality deviates from that expectation across time.

When it is useful:

• quick sanity check that a cohort difference is not noise
• comparing two versions of a system or product feature rollout

When it can mislead:

• if the curves cross (effects change over time)
• if observation windows differ systematically between groups (e.g., one region has shorter follow-up windows)

Cox proportional hazards: turning covariates into “risk multipliers”

The Cox proportional hazards model connects covariates to hazard without specifying the baseline hazard shape. ^[cox-ph]

How to read it in practice:

• $\exp(\beta_j)$ is a hazard ratio for a 1-unit increase in $X_j$.
• Hazard ratio > 1 means “riskier” (event happens sooner on average), < 1 means “protective.”

Toy interpretation (churn).
If your Cox model yields:

• $\exp(\beta_{\text{annual_plan}})=0.7$

Then, holding other covariates fixed, annual-plan users have 30% lower instantaneous churn hazard than monthly-plan users at any given time.

The key assumption: proportional hazards means those hazard ratios are roughly constant over time (i.e., the curves differ by a multiplicative factor in hazard, not by shape). When this is false, you may need:

• time-varying covariates/effects
• stratified Cox
• an accelerated failure time (AFT) model for “time scaling” instead of “hazard scaling” ^{[survival-analysis]}

Practical checklist for engineers

1) Define observation windows explicitly
Write down, in the experiment doc (not just in code), what “duration” means and when observation stops (end of study, export date, decommission time, trace cutoff).

2) Check whether observation stopping is correlated with the event
Ask simple pipeline questions:

• Do some users “disappear” because tracking stops (not because behavior changed)?
• Do some machines leave the dataset because they were proactively replaced (possibly due to warning signs)?

If “stopping observation” is correlated with the event, naive analyses can bias survival upward.

3) Start with KM curves before fitting models
KM gives shape intuition: early drop, long tail, crossing hazards, etc. ^{[kaplan-meier]}

4) Use Cox when you need covariate-adjusted answers
Examples:

• adjust failure risk for load, temperature, and batch
• adjust churn risk for acquisition channel and engagement

5) Operationalize outputs
Good time-to-event analysis ends with a decision:

• “Which cohort should we target, and when?”
• “What burn-in period reduces infant mortality?”
• “Which covariate is the strongest early-warning signal?”

A simple operational trick: evaluate $S(t)$ at business-relevant horizons (e.g., day 1, day 7, day 30) and report those, not just “the curve.”

References

• Survival analysis (overview). ^{[survival-analysis]}
• Kaplan–Meier estimator. ^{[kaplan-meier]}
• Log-rank test. ^{[logrank-test]}
• Proportional hazards / Cox model. ^[cox-ph]
• Introduction to survival data and incomplete observation windows (video). ^{[survival-video]}

In many cases, it’s better to trade a bit more bias for a much smaller variance to tighten generalisation error estimates. One way to do so is by adding a regularizer. In linear regression, L1 (Lasso) and L2 (Ridge) regularization are two common options. Lasso suppresses small weights to zero, making the feature space sparse, while Ridge shrinks all weights. Both restrict the norm of the weights and help mitigate overfitting.

However, regularization is only one way to strike the balance. Another way is to introduce more bias into the equation through the Bayesian lens. Specifically, we can impose prior knowledge by adding a prior distribution to constrain the norm of the leant parameters. For example, if we know the weights are small and centred, then we can set our prior to be $\vec{w} \sim \mathcal{N}(0, \beta\textbf{I})$. Then, by Bayes rule, we have:

Thus, both regularization and Bayesian modelling can achieve the same goal, which begs the question: are they connected?

Laplace is to Lasso as Gaussian is to Ridge

The answer to the above question turns out to be yes! To illustrate this further, let’s use two common prior distributions, Lapace and Gaussian, as our running examples.

Firstly, we assume the following general setting for regression: ${y} = X{\theta}$ and $f_X = y + \epsilon$ where $\theta \sim \text{Laplace}(0, s) = 1/2s \cdot\exp(-\mid\theta\mid / s)$ and $\epsilon \sim \mathcal{N}(0, \delta^2_\epsilon)$.

Then, we obtain the maximum a posteriori (MAP) esitmation as:

Next, we can substitute both the likelihood and prior into Eq. (1).

where $Z$ is the Gaussian normalising constant. By simplifying Eq. (2), we obtain the following form:

Now, we have recovered the exact form of Lasso, where $\cfrac{2\delta^2_\epsilon}{s}$ is the coefficient of the L1 regularizor $\lambda$ that controls the strength of the constraint.

Next, let’s play the same trick in the same setting but with a Gaussian prior instead, i.e., $\theta \sim \mathcal{N}(0, \delta_\theta^2)$.

Starting from Eq. (1), we subsitute in the likelihood and prior as above:

Finally, letting go all the fluff in Eq. (5), we have:

By Eq. (6), we have recovered Ridge regression where the fraction $\cfrac{\delta_{\epsilon}^2}{\delta_{\theta}^2}$ denotes regularization constant $\lambda$.

Summary

By working out the above two examples, we found that regularised regression is nothing but Bayesian modelling in disguise. In fact, imposing various priors has the same effect as using corresponding regularizers. By the same token, choosing different likelihoods gives us different loss functions. In this post, we used Gaussian likelihood in both examples, and, in turn, recovered the square loss.

There are many other options for prior and likelihood functions. For instance, one can use a student-t as opposed to Gaussian. Lastly, a family of conjugate prior can drastically reduce the cost of Bayesian inference.

This is also a rolling log of books I recently read (2021) that are right up my alley.

Bennett, Arnold. Self and self-management: Essays about existing. George H. Doran Company, 1918.

The true stories behind the stories of success shall be the same and might not be as glorious.

Hawking, Stephen. The theory of everything. Jaico Publishing House, 2006.

Well, I’m now under the impression that computer science is a pseudoscience :}

Hall, Herbert James. The untroubled mind. Houghton Mifflin, 1915.

How can I live out a life so fully that worries couldn’t sneak in? Perhaps most importantly, what’s my deeper justification and higher pursuit thereof?

Jones, Diana Wynne, John Sessions, and Stella Paskins. Howl’s moving castle. Recorded Books, 2008.

I finally understand why I didn’t quite understand the movie 😅

Some of my friends from China saw a bunch of blanks here. This is because YouTube videos can’t pass the firewall 🧱.

10 ways to have a better conversation

• If you want to pontificate, go write a blog.
• Ask open-ended questions.
• “No man ever listened his way out of a job” — Calvin Coolidge
• Most of us don’t listen with the intent to understand. We listen with the intent to reply.
• Listen, and be prepared to be amazed.
• If you don’t know, say that you don’t know.
• Don’t equate your experience with others.

Self-control

This Ted talk conveys very similar messages as that of the below one. Their main ideas are covered in many other talks as well.

To reach beyond your limits by training your mind

• Self-control is the problem where we have all these desires from ourselves for the long-term, but then in the short-term, we do rather different things (that prevent ourselves from achieving long-term those goals).
• Our will power is weak, and therefore, it’s not something upon which we should rely during decision-making.
• If we are faced with temptation whilst having no tool at hand to overcome it, we’re almost certainly going to fail.
• We should collaborate with our brains with constructive messages.
  • Change the pictures and the words. Using very detailed words.
  • Tell your mind exactly what you want.
• We ought to create tools that will control our future selves to do what our current selves want them to do.
  • It’s a situation where we know we will be tempted, and we do something to make ourselves not be able to be tempted.
• Make the familiar unfamiliar and the unfamiliar familiar.
• Reward-substitution: connecting pain to pleasure
  • Link massive pleasure to going there and pain to not going there.
  • Do the right thing for the wrong reason.
• Make Self Belief so normal to you that everyone believes in you too.

The last point ties in with another Ted talk, which will be introduced later.

Body language, the power is in the palm of your hands

• The palm up position is friendly and inviting, whilst the palm down position exerts power and control over others.
  • Finger pointing is the worst; it is directive and rude.

Imposters: The psychology of pretending to be someone you’re not

• Some imposters are escapologists — they’re running away from a flawed past and trying to rehabilitate their image.
• We spend a hell of a lot of time trying to impress other people.
• Ultimately, life is a performance with all the sense of drama, anarchy, and possibility.
• Who ever controls the past controls the future; who ever controls the present controls the past. — Orwell

How to draw to remember more

• Thinking in pictures and then drawing them down is a great way to learn and memorize new things.
• There are a myriad of ways to represent an abstract concept using drawings.
• The quality of the drawing does not matter at all. In other words, good and bad pictures have rather similar, if not the same, effect on the learning process.
• It is the process of doing the drawing that actually makes a difference.

To be continued 👨‍💻 …

Step 1: key pair generation

Use the following command to generate a public (silver) / private (black) RSA key pair under the ~/.ssh/id_rsa directory. The .ssh/id_rsa is the private key you keep on your machine, and .ssh/id_rsa.pub is the public key you distribute to other machines to enable passwordless logins.

ssh-keygen -t rsa

Note that git uses a different type of cryptosystem, namely, the Ed25519 system, which can be generated using the following command.

ssh-keygen -t ed25519 -C "youremail@yourdomain"

Compared to RSA, it is considered to be faster, safer and more compact (Ed25519: 8chars, RSA: 544chars) although RSA is more commonly used.

Step 2: distribute key(s)

If Alice wants to log in Server1 shown in the figure, she can run the following command to forward her SSH public key(s) to it. The keys sent will be recorded in .ssh/authorized_keys of the host.

ssh-copy-id alice-username@server1.domain-or-ip

Afterwards, Alice should be able to log in Server1 without being asked for her password. I.e.,

ssh alice-username@server1.domain-or-ip

Welcome to XXX ...

Note that for Mac users that do not have ssh-copy-id, you can either intall it via brew or mannually copy these ssh files through scp, rsync or whatnot. If you go for the latter, one thing you should keep in mind is to set the permission bits correctly as shown below.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/*

Step 3: indirect login

In the case that Server1 has a firewall or what have you, Alice has to connect it via a proxy, say Server2; therein lies the question: how to access Server2 directly using key-pair authentication?

To tackle this, Alice can first forward her/his keys to Server2, the proxy, through Step 2. Next, Alice should log in Server2 to generate a key pair (Step 1), and then, send keys to Server1.

Last, but certainly not least, Alice should set up her ssh on her local machine. The configuration (~/.ssh/config) is along the lines of the following.

Host server2
    HostName          server1-domain
    User              alice-username
    IdentityFile      ~/.ssh/id_rsa

Host server1
    HostName          server1-domain
    User              alice-username
    ForwardX11Trusted yes
    ForwardAgent      yes
    IdentityFile      ~/.ssh/id_rsa
    ProxyCommand ssh server2 -W %h:%p

Now, everything should be in place. Alice should be able to do login, port forwarding, or whatnot, with automatic key-pair authentication (without having to type her password every single time for every single server along the way!).

🛑 NB: Data loads between Server1 and Server2 is not encrypted.

# Log in *Server1* via *Server2*.
ssh server1 

# Tunnelling ports to *Server1* via *Server2*.
ssh -NL {listening_port}:{hostmachine}:{host_port} server1 

p.s. Wrestling with company proxies during these COVID times we live in at the moment can be devastating 😷