This lecture introduces the concept of probability and basic probability laws. Applications are deferred to next lecture.

There are two types of lecturers: one that follows strictly a plan, being it in the form of Powerpoint slides, lecture notes, or a textbook, and the other jumping around, even though they had a plan. Apparently, I belonged to the second kind. The only plan that I followed was the following, informal definition of probability:

Probability is a measure of how likely it is that something will happen, or that a statement is true.

This teaching approach requires students to do their own readings. To me, learning involves two processes: private learning and absorbing; and discussions and debates. The purpose of discussions is to help integrate the new knowledge into our own existing knowledge network – a process of Bayesian updating.

Back to the informal definition of probability given above, I forgot where I took this definition – likely from Wikipedia – but this definition fits my purpose of emphasizing the double faces of probability: that is, a probability model can be applied to both a natural system (e.g., a building, a transport network), which is objective, and a decision system (e.g., a design process, an asset management planning process).

I provided students with a lecture note. The note includes a section “What is probability?” to explain various interpretations of probability. Although there are at least six different interpretations, my notes focus on only the classical, frequentistic, and Bayesian interpretations. I decided not to discuss this during my lecture. They are interesting and hard topics. A thorough discussion of these interpretations will require more than 3 hours, which I cannot afford. On the other hand, not every student is philosophically oriented. From applications point of view, most of students probably just want to know which stands you are taking and why. This is probably a safe approach.

My selection is a Bayesian one. From a pragmatic view, a coin to be tossed and a coin that’s covered on the desk are, to me (a modeller), is no difference. They both can be modelled by a probability model. Unless there is a way (or we can afford to take the way) to collect all necessary data for the coin dynamics from the start of tossing to the landing, these two coins can take the same probability model. From a theoretic view, separating probability into frequentistic probability and subjective (or Bayesian) probability is a fact of history of knowledge discovery, but we do not need to fall into the trap of history. All knowledge is relative; using the jargon of probability, conditional! All theories has a boundary. Therefore, a full discourse of any serious matter must take the background knowledge (or assumptions) into consideration. This is the essence of risk analysis, and this is the major reason that we do not use ‘risk-based’ decision making, but rather ‘risk-informed’ decision making. Separating probability into subjective and objective probabilities will create ourselves a new trap: what is the sum (or product) of an apple and an orange?

Suppose the coin to be tossed is modelled by an ‘objective’ probability, and the coin being covered on the desk by a ‘subjective’ probability. If either of the coin is on heads, take 1, otherwise 0. Ask: what is the probability that the sum of the two coins will be 1? There are three possibilities: 0 (TT), 1 (HT & TH), and 2 (HH). The answer is 50%. Is it similar to like adding an apple to an orange?

Why do we need to formalize the definition of probability?

Andre Kolmogorov axiomatized the probability theory. As an engineering student, do we really need to care the theory?

I provided in the lecture two motivational examples. One is taken Khaneman’s Thinking – Fast and Slow:

Linda is thirty-one years old, single, outspoken, and very bright.  She majored in philosophy.  As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in antinuclear demonstrations.

Does Linda look more like

• A. A bank teller,
• B. An insurance salesperson, or
• C. A bank teller active in the feminist movement?

This example shows the significance of following probability laws if daily reasoning.

The second motivational example is the famous Bertrand’s paradox:

A random chord is drawn on the unit circle.  What is the probability that its length exceeds √3, the length of the side of the equilateral triangle inscribed in the unit circle?

What do we mean by randomness? Three different definitions of ‘a random chord’ give three different estimation of the probability: 1/2, 1/3, and 1/4. Amazing!

Every Probability is Conditional!

Conditional probability of A given B is defined as Pr(A|B) = Pr(AB)/Pr(B). The event B becomes a normalizer. Based on this, we can take Pr(A) as Pr(A|O), where O here represents the sample space $\Omega$ (thanks TMU’s wordpress, it doesn’t support Latex writing!).

Two events A and B are said to be independent of each other if Pr(A|B) = Pr(A), or Pr(B|A) = Pr(B), or Pr(AB) = Pr(A)xPr(B). In daily language, the independence basically means that knowing a new information B does not change your estimation (or belief) of the probability of A.

A common confusion that beginners often have is between the notion of mutual exclusion and statistical dependence. Events A and B are mutually exclusive if Pr(AB) = 0. In the Venn’s diagram, A and B have no overlapped area. So mutual exclusion can be easily visualized. Independence is very difficult to visualize, although an area analogy to probability may be helpful – that is, think of Pr(A) being the proportion of the whole area of the sample space, Area(A)/Area(O); then Pr(A|B) becomes Area(AB)/Area(B). If the two proportion of A to Omega happens to the same as the proportion of AB to B, then A and B are independent.

Conditional Independence and Bayesian Network

Two events A and B are said to be conditional independent of each other given C, if Pr(AB|C) = Pr(A|C) Pr(B|C).

The conditional independence is a powerful assumption for dealing with complex joint events. A special case is the Markov property:

Pr(X₁ X₂ … X_n) = Pr(X₁) Pr(X₂|X₁) Pr(X₃|X₂) … Pr(X_n|X_n-1)

A modern application of the conditional independence is through the notion of Bayesian network, which will be explained in the next or further next lecture.

Total Probability Formula

The total probability formula is a powerful tool in probabilistic risk analysis. The formula looks naively simple:

Pr(A) = Pr(AE₁) + … + Pr(AE_n) = Pr(A|E₁) Pr(E₁) + … + Pr(A|E_n) Pr(E_n)

In real-world problems, there may be multiple, mutually exclusive, ways to trigger an event A. These exhaustive, mutually exclusive events are denoted as E1, …, En (sometimes referred to as initiating events). Then the probability of A can be evaluated separately using the old ‘divide-and-conquer’ wisdom.

Bayes Rule

Many people underestimate the value of the Bayes formula. This time, I introduced it by introducing a toy problem first:

A city is served by three overnight mail carriers called A, B, and C.  The past record indicates that they fail to deliver the mail on time 1%, 2%, and 3% of the time, respectively.  If the overnight letter arrived late, what is the probability that it was sent via A? via B? via C?

The background information was intentionally missing. This aligns with the historical development of the so-called inverse probability problem. Knowing Pr(A|B), how do we find Pr(B|A)?

The value of the Bayes formula is not the mathematical formula itself, but lies on the interpretations of Pr(A) and Pr(A|E), with E representing the new ‘evidence’ and A the event of major interest.

With new evidence E arriving, how do we update our belief of A? There are two layers of issues here. The first layer is that A and E must be dependent somehow. If they are independent of each other, the new evidence does not provide any new information about A, and hence there is no updating.

The second layer of significance is the ‘outside of box’, or systems viewpoint. To complete the updating, we need to look at not only how A would affect the occurrence of E, but also other ‘reasons’ that may trigger the occurrence of E as well. Only through this holistic assessment, can one properly update their belief on A.

First Lecture, September 9, Overview

Leaves of the honey locusts outside of my office have turned light yellow. The fall is here.

As always, the first lecture needs to first give students an overview of the subject, and then tell students the focus of the course. Nobody can cover all the topics of a subject. Selecting the topics to be taught in the next 11 or 12 sessions is not an easy task. When you are new in the subject, you may struggle to make up the full course. But when you have more than the time allows, you have to cut. And cutting hurts.

Last night Amazon finally delivered the copy of Der Kiureghian (2020) Structural and System Reliability to my doorsteps. I had a quick scan on the way to school. It is a great text, but I still hesitate to use it as the text for this course. It can be a good one for structural engineering students. But my class has probably only a half of them coming from structural/geotechnical engineering, and the other half from transportation, environmental and construction management.

Reliability itself may need two courses; one for structural reliability, the other for engineering reliability. Structural reliability takes a deductive approach, starting from random variables and their functions to reliability-based design. Monte Carlo simulation, surrogate modelling, code calibration and development, load combinations, and reliability evaluation of existing structures can be decently covered. Engineering reliability takes a inductive approach, starting from reliability or failure data analysis through stochastic deterioration modelling to inspection and maintenance optimization. There is no single textbook to my knowledge that covers both structural reliability and engineering reliability. Civil engineers still don’t talk to mechanical or electrical engineers. Can I teach students to build this bridge? I tried a few times, not successful. Students only complained that it was a hard course. Sad!

I did a pre-course survey through D2L last week. One question asks students to put down their learning expectations. One student put: “I hope to learn about Canadian laws and building codes.” I used to put ‘Beyond great expectations’ as the subtitle of the first lecture slide, but this expectation is huge! It is not irrelevant. I will try to entice some students to the code comparisons (e.g., NBCC vs. ASCE 7, AASHTO vs. CHBDC). Laws, ultimately, are one of the most effective framework for managing risks. At one time, I thought of initiating a private research project matching construction contracts with construction project risks. So far, this idea still stays as only an idea. So, I am sorry, this student; I won’t be able to cover Laws.

Since the word reliability has already mad you so busy, why bother to introduce risk?

Reliability is an abstract concept. In structural reliability, the concept arises from the need of defining safety and serviceability. Most of us can agree that there is no absolute safety. So we need a measure to define the relative safety so that it can help address the ‘how safe is safe enough’ question. Probability is found to be a convenient, existing mathematical notion. Hence, reliability denotes the probability of no failure, or 1 minus failure probability. Neglect for now all the analysis details, which Prof. Der Kiureghian takes the whole book to explain. But what is the meaning of probability? Can we validate the calculated probability of failure empirically? If not, is structural reliability a scientific theory?

This brings the subjective interpretation of probability. The whole structural reliability community takes this interpretation. This tradition starts from Prof. Cornell (see the preface of his 1971 textbook). In a recent article for JCSS 50 Interpretation of probability in structural safety – A philosophical conundrum“, Ton Vrouwenvelder et al. (2024) reiterated this Bayesian, subjective stand.

Students won’t fully buy this in until they were presented this question in a decision-making setting. Past engineering education places too much emphasis on the scientific foundations of engineering, and yet little discussions about decision making has been given to students. In my view, introducing the basic setting and categorization of engineering decisions a good start of the subject.

A decision involves three basic elements: alternatives or options, outcomes, and criterion or criteria. To decide the best option to take in front of a decision, one needs to list all meaningful options they may have, obtain the outcome of every option, and know how to compare the outcomes based on the decision maker’s value system. Decision theory focuses on the last question, i.e., how to compare outcomes in order to conclude the decision. This seems to be an easy task, but it is actually not. There are two basic categories:

• Multi-criteria decision making
• Decision making under uncertainty

In the first category, the outcomes of a decision alternative involve multiple aspects. For example, when we buy a new car, we need to compare multiple models. In this comparison, may look at not only the price, but also the power, fuel consumption, safety, aesthetics, and so on. Some of the attributes can be converted to a standard unit. For example, the purchase price and the fuel consumption may be lumped into a lifecycle cost if we can properly estimate the expected service life. However, apparently not all attributes can be converted to a unified measure. This brings the decision maker to a difficult situation where they have to compare ‘apple and orange.’

The decision making under uncertainty is different. In this case, the outcome of a decision alternative is uncertain. In general, it can be described as a random variable with a given probability distribution. The question thus turns to the comparison of two random variables. In the simplest case, we are facing a decision like the following between two options:

• Option 1: The net benefit is zero.
• Option 2: The net benefit is $5000 with 20% and -$1000 with 80%.

Do you prefer Option 1 or 2?

At the core, decision theory analyzes the coherence of decision rules. For example, do we take the option based on the mean value (in probability, it is also called expected value, or expectation) of the random outcome? Does the mean criterion provide a coherent decision? Does the variance or standard deviation matter? How do we settle down the conservatism embedded in engineering? Pushing it to limit, we often face hard engineering decisions of “low probability, large consequences.” That’s why we need to know a little decision theory. More importantly, the statistical decision theory takes probability purely as the machinery of logic. It does not matter whether the probability is interpreted as an empirical frequency, an inherent nature of the subject under study, or a degree of belief of the decision maker on the subject. Once we understand this, we can settle down the struggle, for most of the part, whether we need to separate aleatory and epistemic uncertainties.

This course focuses on decision making under uncertainty, in which the uncertainty can be modelled using probability theory. We will not discuss game theory, which deals with another DMUU situation where a decision involves multiple opponents and the uncertainty is primarily caused by the opponents’ decisions.

Engineering decisions add two complications to the DMUU problems: First, the number of decision alternatives can be huge or even infinite. Second, the outcome analysis of each alternative often involves complicated system analysis. Mathematical optimization is used to address the first aspect, whereas probabilistic system analysis, or simply, reliability analysis, is used to address the second aspect.

Broadly speaking, reliability analysis addresses two major problems:

1. Characterization and quantification of uncertainties
2. Propagation of uncertainties from inputs to outputs

In terms of uncertainty characterization, there are probability models, random variable models, random vector models, stochastic process models, and random field models. For each model, we need to understand the full and partial characterization. For example, for a random variable model, the full characterization requires the determination of its probability distribution. But it can partially characterized by its mean, standard deviation, skewness, kurtosis, etc. How do the full and partial characterizations related to each other? This is also an interesting question. Understanding this will help you understand the various reliability methods and their relationships.

We need empirical data to quantify the uncertainty. This involves the study of statistics. Likelihood function is the bridge between probability and statistics. It plays the pivotal role in Bayesian statistics as well. Therefore, we will focus on the construction of likelihood functions for various models. Reliability data often involves missing and incomplete information. The likelihood function provides a proper mechanism to address the data imperfections as well.

Regarding uncertainty propagation, we shall study both analytical methods and simulation based methods.

If time permits, we will introduce Response Surface as the prelude to surrogate modelling for complicated system analysis.

Various application areas will be largely left for students to explore by themselves through their course projects.