Regression model
Binary logit/probit model
P ( y = 1 ) = P ( ∑ β x + ϵ > 0 ) = P ( ϵ > − ∑ β x ) = 1 − F ( − ∑ β x ) = F ( ∑ β x ) \begin{aligned}
P(y=1) &= P(\sum \beta x + \epsilon > 0) \\
&= P(\epsilon > -\sum \beta x) \\
&= 1 - F(-\sum \beta x) \\
&= F(\sum \beta x)
\end{aligned} P ( y = 1 ) = P ( ∑ β x + ϵ > 0 ) = P ( ϵ > − ∑ β x ) = 1 − F ( − ∑ β x ) = F ( ∑ β x )
logit CDF: F ( x ) = e x 1 + e x F(x)=\frac{e^x}{1+e^x} F ( x ) = 1 + e x e x
probit CDF: F ( x ) = ϕ ( x ) = ∫ − ∞ x 1 2 π e − z 2 2 d z F(x)=\phi(x)=\int ^{x} _{-\infty}{ \frac{1}{ \sqrt{2 \pi} } e^{\frac{-z^2}{2}}}dz F ( x ) = ϕ ( x ) = ∫ − ∞ x √ 2 π 1 e 2 − z 2 d z
marginal effect
logit
odds is P 1 − P = e ∑ β x \frac{P}{1-P}=e^{\sum \beta x} 1 − P P = e ∑ β x
odds ratio is o d d s ( a + 1 ) o d d s ( a ) = e β x \frac{odds(a+1)}{odds(a)}=e^{\beta x} o d d s ( a ) o d d s ( a + 1 ) = e β x
probit
Φ − 1 ( p ) = ∑ β x \Phi^{-1}(p) = \sum \beta x Φ − 1 ( p ) = ∑ β x
Φ a = 1 − 1 − Φ a = 0 − 1 = β a \Phi^{-1}_{a=1} - \Phi^{-1}_{a=0} = \beta a Φ a = 1 − 1 − Φ a = 0 − 1 = β a
Order logit/probit model
Dependant variables are ordered values((ex) worst worse normal good very good)
P ( y = 1 ) = P ( y ∗ ≤ 0 ) = P ( ε ≤ − ∑ β x ) = F ( − ∑ β x ) P ( y = 2 ) = P ( 0 ≤ y ≤ μ 2 ) = P ( − ∑ β x < ε < μ 2 − ∑ β x ) = F ( μ 2 − ∑ β x ) − F ( − ∑ β x ) \begin{aligned}
P(y=1) &= P(y^* \leq 0) = P(\varepsilon \leq -\sum \beta x) = F( -\sum \beta x) \\
P(y=2) &= P(0 \leq y \leq \mu_2) = P(-\sum \beta x < \varepsilon < \mu_2 - \sum \beta x) = F (\mu_2 - \sum \beta x) - F(-\sum \beta x)
\end{aligned} P ( y = 1 ) P ( y = 2 ) = P ( y ∗ ≤ 0 ) = P ( ε ≤ − ∑ β x ) = F ( − ∑ β x ) = P ( 0 ≤ y ≤ μ 2 ) = P ( − ∑ β x < ε < μ 2 − ∑ β x ) = F ( μ 2 − ∑ β x ) − F ( − ∑ β x )
marginal effect
logit
odds is P ( y = 1 ) P ( y ≠ 1 ) \frac{P(y=1)}{P(y \neq 1)} P ( y ≠ 1 ) P ( y = 1 )
odds ratio is o d d s ( a = 1 ) o d d s ( a = 0 ) = e β a \frac{odds(a=1)}{odds(a=0)}=e^{\beta a} o d d s ( a = 0 ) o d d s ( a = 1 ) = e β a
probit
Φ − 1 ( p ( y = 1 ) ) = z \Phi^{-1}(p(y=1)) = z Φ − 1 ( p ( y = 1 ) ) = z
Φ a = 1 − 1 − Φ a = 0 − 1 = β a \Phi^{-1}_{a=1} - \Phi^{-1}_{a=0} = \beta a Φ a = 1 − 1 − Φ a = 0 − 1 = β a
boundary value
logit
odds is P ( y ≤ i ) P ( y > i ) \frac{P(y\leq i)}{P(y > i)} P ( y > i ) P ( y ≤ i )
odds ratio is o d d s ( a = 1 ) o d d s ( a = 0 ) = e μ i − β a \frac{odds(a=1)}{odds(a=0)}=e^{\mu i - \beta a} o d d s ( a = 0 ) o d d s ( a = 1 ) = e μ i − β a
probit
Φ − 1 ( P ( y ≤ i ) ) − Φ − 1 ( P ( y ≤ i − 1 ) ) = μ i \Phi^{-1}(P(y \leq i)) - \Phi^{-1}(P( y \leq i-1)) = \mu i Φ − 1 ( P ( y ≤ i ) ) − Φ − 1 ( P ( y ≤ i − 1 ) ) = μ i
Multinominal logit model
In case of the number of independent variables is greater than 3.
p j p j + p J = F ( ∑ k β j k x k ) \frac{p_j}{p_j + p_J} = F(\sum_{k} \beta_{jk} x_k)
p j + p J p j = F ( k ∑ β j k x k )
where p_J is a reference variable and p j = j v a r i a b l e p J + j v a r i a b l e p_j = \frac{\text{j variable}}{p_J + \text{j variable}} p j = p J + j v a r i a b l e j v a r i a b l e
p j p J = e ∑ β j k x k \frac{p_j}{p_J}=e^{\sum \beta_{jk} x_k}
p J p j = e ∑ β j k x k
∑ j p j p J = 1 − p j p J = 1 p J − 1 = ∑ i e β j k x k \sum_j \frac{p_j}{p_J} = \frac{1-p_j}{p_J}=\frac{1}{p_J}-1=\sum_i e^{\beta_{jk}x_k}
j ∑ p J p j = p J 1 − p j = p J 1 − 1 = i ∑ e β j k x k
p J = 1 1 + ∑ j e ∑ β j k x k p_J=\frac{1}{1 + \sum_j e^{\sum \beta_{jk} x_k}}
p J = 1 + ∑ j e ∑ β j k x k 1
p j = e ∑ β j k x k 1 + ∑ j e ∑ β j k x k p_j=\frac{e^{\sum \beta_{jk} x_k}}{1 + \sum_j e^{\sum \beta_{jk} x_k}}
p j = 1 + ∑ j e ∑ β j k x k e ∑ β j k x k
Nested logit/probit model
independent variable is hierarchy structure
buy car
used
new
no buy car
used
new
F
F_1
1-F_1
F_k12
1-F_k12
f_k22
1-f_k22
marginal effect
logit
odds is F k i 1 − F k i \frac{F_{ki}}{1-F_{ki}} 1 − F k i F k i
odds ratio is o d d s ( a = 1 ) o d d s ( a = 0 = β a \frac{odds(a=1)}{odds(a=0}=\beta a o d d s ( a = 0 o d d s ( a = 1 ) = β a
probit
Φ − 1 ( P ( y = 1 ) ) = ∑ β x \Phi^{-1}(P(y = 1)) = \sum \beta x Φ − 1 ( P ( y = 1 ) ) = ∑ β x
Φ a = 1 − 1 − Φ a = 0 − 1 = β a \Phi^{-1}_{a=1} - \Phi^{-1}_{a=0} = \beta a Φ a = 1 − 1 − Φ a = 0 − 1 = β a
Conditional model
In case independent variables is changed by depedent variables
ex: independent variables -> cost, time by seoul, gyunggi
dependent variables -> bus train car
p ( b u s ) = e b u s e t r a i n + e c a r + e b u s p(\text{bus})=\frac{e^{\text{bus}}}{e^{\text{train}}+e^{\text{car}}+e^{\text{bus}}}
p ( b u s ) = e t r a i n + e c a r + e b u s e b u s
e β a e^{\beta a} e β a : when a variable increase 1, the increase ratio reference variable(car) to comparable variable(bus)
marginal effect
∂ P j ∗ ∂ z j = ∂ ∂ z j k e ∑ a z ∑ e a z ( Q = ∑ e a z ) = α e ∑ e a z Q − α e 2 ∑ e a z Q 2 = α e ∑ α z Q ( Q − e e ∑ α z Q ) = α k P j ( 1 − P j ) ( P = e ∑ α z Q ) \begin{aligned}\frac{\partial P_{j^*}}{\partial z_{j}}&=\frac{\partial}{\partial z_{jk} } \frac{e^{\sum a z}}{\sum e^{az}} \quad (Q=\sum e^{az})\\
&=\frac{\alpha e^{\sum e^{az}} Q - \alpha e^{2 \sum e^{az}} }{Q^2} \\
&=\frac{ \alpha e^{\sum \alpha z}}{Q} (\frac{Q - e^{e^{\sum \alpha z}}}{Q}) \\
&= \alpha_k P_j(1- P_j) \quad (P =\frac{ e^{\sum \alpha z}}{Q} ) \end{aligned} ∂ z j ∂ P j ∗ = ∂ z j k ∂ ∑ e a z e ∑ a z ( Q = ∑ e a z ) = Q 2 α e ∑ e a z Q − α e 2 ∑ e a z = Q α e ∑ α z ( Q Q − e e ∑ α z ) = α k P j ( 1 − P j ) ( P = Q e ∑ α z )
Ranked logit model
In case of ranked dependent variables
Marginal effect
e β a = λ λ 0 λ : H a z a r d f u n c t i o n e^{\beta a} = \frac{\lambda}{\lambda_0} \quad \lambda \text{: Hazard function} e β a = λ 0 λ λ : H a z a r d f u n c t i o n
probability
P ( u r 1 > u r 2 … ) = ∏ j = 1 J − 1 e V j ∑ e V k \displaystyle P(u_{r_1} > u_{r_2} \ldots)=\prod_{j=1}^{J-1} \frac{ e^{V_j}}{\sum e^{V_k}} P ( u r 1 > u r 2 … ) = j = 1 ∏ J − 1 ∑ e V k e V j
where V j = β j x i + α z j + θ w i j V_j = \beta_j x_i + \alpha z_j + \theta w_{ij} V j = β j x i + α z j + θ w i j
rank
exp ( ∑ i ≠ α β i x i + α ∑ z j ) \exp(\sum_{i \neq \alpha} \beta_i x_i + \alpha \sum z_j) exp ( ∑ i ≠ α β i x i + α ∑ z j )
where α \alpha α is means, z j z_j z j
Example probability
the probability of a(1st) b(sec) c(thd):
a ⋅ b ⋅ c = exp ( P ( a ) ) exp ( a ) + exp ( b ) + exp ( c ) ⋅ exp ( P ( b ) ) exp ( b ) + exp ( c ) ⋅ exp ( P ( c ) ) exp ( c ) a \cdotp b \cdotp c =\frac{\exp(P(a))}{\exp(a)+\exp(b)+\exp(c)} \cdotp \frac{\exp(P(b))}{\exp(b)+\exp(c)}\cdotp \frac{\exp(P(c))}{\exp(c)} a ⋅ b ⋅ c = exp ( a ) + exp ( b ) + exp ( c ) exp ( P ( a ) ) ⋅ exp ( b ) + exp ( c ) exp ( P ( b ) ) ⋅ exp ( c ) exp ( P ( c ) )