1、. . . . . . . . . . . . . . . . . . . . . . . . . . . .24 6.3A Stick-breaking Approach to IBP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 1Introduction to the Dirichlet Distribution An example of a pmf is an ordinary six-sided die - to sample the pmf you roll the die and prod
2、uce a number from one to six. But real dice are not exactly uniformly weighted, due to the laws of physics and the reality of manufacturing. A bag of 100 real dice is an example of a random pmf - to sample this random pmf you put your hand in the bag and draw out a die, that is, you draw a pmf. A ba
3、g of dice manufactured using a crude process 100 years ago will likely have probabilities that deviate wildly from the uniform pmf, whereas a bag of state-of-the-art dice used by Las Vegas casinos may have barely perceptible imperfections. We can model the randomness of pmfs with the Dirichlet distr
4、ibution. One application area where the Dirichlet has proved to be particularly useful is in modeling the distribu- tion of words in text documents 9. If we have a dictionary containing k possible words, then a particular document can be represented by a pmf of length k produced by normalizing the e
5、mpirical frequency of its words. A group of documents produces a collection of pmfs, and we can fi t a Dirichlet distribution to capture the variability of these pmfs. Diff erent Dirichlet distributions can be used to model documents by diff erent authors or documents on diff erent topics. In this s
6、ection, we describe the Dirichlet distribution and some of its properties. In Sections 1.2 and 1.4, we illustrate common modeling scenarios in which the Dirichlet is frequently used: fi rst, as a conjugate prior for the multinomial distribution in Bayesian statistics, and second, in the context of t
7、he compound Dirichlet (a.k.a. P olya distribution), which fi nds extensive use in machine learning and natural language processing. Then, in Section 2, we discuss how to generate realizations from the Dirichlet using three methods: urn-drawing, stick-breaking, and transforming Gamma random variables
8、. In Sections 3 and 6, we delve into Bayesian non-parametric statistics, introducing the Dirichlet process, the Chinese restaurant process, and the Indian buff et process. 1.1 Defi nition of the Dirichlet Distribution A pmf with k components lies on the (k 1)-dimensional probability simplex, which i
9、s a surface in Rk denoted by k and defi ned to be the set of vectors whose k components are non-negative and sum to 1, that is k= q Rk| Pk i=1qi = 1,qi 0 for i = 1,2,.,k. While the set klies in a k-dimensional space, k is itself a (k 1)-dimensional object. As an example, Fig. 1 shows the two-dimensi
10、onal probability simplex for k = 3 events lying in three-dimensional Euclidean space. Each point q in the simplex can be thought of as a probability mass function in its own right. This is because each component of q is non-negative, and the components sum to 1. The Dirichlet distribution can be tho
11、ught of as a probability distribution over the (k 1)-dimensional probability simplex k; that is, as a distribution over pmfs of length k. Dirichlet distribution:Let Q = Q1,Q2,.,Qk be a random pmf, that is Qi 0 for i = 1,2,.,k and Pk i=1Qi = 1. In addition, suppose that = 1,2,.,k, with i 0 for each i
12、, and let 0= Pk i=1i. Then, Q is said to have a Dirichlet distribution with parameter , which we denote by Q Dir(), if it has1 f(q;) = 0 if q is not a pmf, and if q is a pmf then f(q;) = (0) Qk i=1(i) k Y i=1 qi1 i ,(1) 1The density of the Dirichlet is positive only on the simplex, which as noted pr
13、eviously, is a (k 1)-dimensional object living in a k-dimensional space. Because the density must satisfy P(Q A) = R Af(q;)d(q) for some measure , we must restrict the measure to being over a (k 1)-dimensional space; otherwise, integrating over a (k 1)-dimensional subset of a k-dimensional space wil
14、l always give an integral of 0. Furthermore, to have a=(儀匀罚儁儁儀讀缁弋H缀栞礄圀椀褂嬃霃鞃霃攃罣噞晒搀漀挀愀昀搀攀攀挀昀戀攀攀愀攀最椀昀罣噞晒搀漀挀尀尀戀搀攀挀搀搀愀挀搀挀刀挀樀吀焀伀昀夀堀洀圀猀夀稀攀最夀礀礀攀瘀洀挀愀夀琀倀吀瘀夀稀儀罣嘀愀挀挀昀昀愀搀搀:栀艹艹艹u艹葠絶晹攀琀眀娀夀栀吀刀昀砀渀最夀唀儀匀砀爀娀爀稀焀洀一礀搀猀砀氀瘀洀漀栀瘀欀漀欀儀焀最罣噞晒噧晒葨罣鑞畞 攠臿羉祏r葓鹎啓f葭筟膘媍潧綂葙啎祏齢Sf葭罣扜扜珿罓豻呜疂耀舀(磬. . . . . . . . . . . . . . . . . . . . . . . .
15、. . . .24 6.3A Stick-breaking Approach to IBP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 1Introduction to the Dirichlet Distribution An example of a pmf is an ordinary six-sided die - to sample the pmf you roll the die and produce a number from one to six. But real dice are
16、not exactly uniformly weighted, due to the laws of physics and the reality of manufacturing. A bag of 100 real dice is an example of a random pmf - to sample this random pmf you put your hand in the bag and draw out a die, that is, you draw a pmf. A bag of dice manufactured using a crude process 100
17、 years ago will likely have probabilities that deviate wildly from the uniform pmf, whereas a bag of state-of-the-art dice used by Las Vegas casinos may have barely perceptible imperfections. We can model the randomness of pmfs with the Dirichlet distribution. One application area where the Dirichle
18、t has proved to be particularly useful is in modeling the distribu- tion of words in text documents 9. If we have a dictionary containing k possible words, then a particular document can be represented by a pmf of length k produced by normalizing the empirical frequency of its words. A group of docu
19、ments produces a collection of pmfs, and we can fi t a Dirichlet distribution to capture the variability of these pmfs. Diff erent Dirichlet distributions can be used to model documents by diff erent authors or documents on diff erent topics. In this section, we describe the Dirichlet distribution a
20、nd some of its properties. In Sections 1.2 and 1.4, we illustrate common modeling scenarios in which the Dirichlet is frequently used: fi rst, as a conjugate prior for the multinomial distribution in Bayesian statistics, and second, in the context of the compound Dirichlet (a.k.a. P olya distributio
21、n), which fi nds extensive use in machine learning and natural language processing. Then, in Section 2, we discuss how to generate realizations from the Dirichlet using three methods: urn-drawing, stick-breaking, and transforming Gamma random variables. In Sections 3 and 6, we delve into Bayesian no
22、n-parametric statistics, introducing the Dirichlet process, the Chinese restaurant process, and the Indian buff et process. 1.1 Defi nition of the Dirichlet Distribution A pmf with k components lies on the (k 1)-dimensional probability simplex, which is a surface in Rk denoted by k and defi ned to b
23、e the set of vectors whose k components are non-negative and sum to 1, that is k= q Rk| Pk i=1qi = 1,qi 0 for i = 1,2,.,k. While the set klies in a k-dimensional space, k is itself a (k 1)-dimensional object. As an example, Fig. 1 shows the two-dimensional probability simplex for k = 3 events lying
24、in three-dimensional Euclidean space. Each point q in the simplex can be thought of as a probability mass function in it=m(儀匀汀缸儁儁儀讀缁%H缀渀翺堀椀脂漃澃漃礃贃锃锃锃暕颋癓呑搀漀挀挀搀攀挀挀昀挀戀搀搀搀攀愀最椀昀暕颋癓呑搀漀挀尀尀戀攀昀戀挀愀愀愀挀稀昀匀刀琀爀攀猀昀焀最攀夀圀砀稀砀欀礀栀戀甀夀琀稀愀儀椀嘀猀夀洀欀昀搀颋癓吀搀愀攀攀搀挀攀昀戀挀攀缀栀艹6艹葠絶晹儀砀焀堀搀甀琀瀀唀甀爀攀氀樀倀眀欀稀欀栀夀渀搀砀渀戀夀娀猀瘀焀吀礀戀搀爀稀圀暕瞋颀呖呗须瞋詟瑢昰歔饘呗须瞋煓
25、甀殘顙畿形畣彾颋呓须琀昀頀谀鐀洀欀匀疂耀舀(磬. . . . . . . . . . . . . . . . . . . . . . . . . . . .24 6.3A Stick-breaking Approach to IBP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 1Introduction to the Dirichlet Distribution An example of a pmf is an ordinary six-sided die - to sample the pmf yo
26、u roll the die and produce a number from one to six. But real dice are not exactly uniformly weighted, due to the laws of physics and the reality of manufacturing. A bag of 100 real dice is an example of a random pmf - to sample this random pmf you put your hand in the bag and draw out a die, that i
27、s, you draw a pmf. A bag of dice manufactured using a crude process 100 years ag=r(儀匀罚儁儁儀讀缁朰H缀穒栞礋圀椀匂夃夃夃餃欃欜欜甜褜鄜鄜鄜螋葎豶啓膗蒏蝎搀漀挀愀昀搀戀攀戀戀挀愀昀最椀昀螋葎豶啓膗蒏蝎搀漀挀尀尀戀攀昀愀攀愀愀戀愀攀愀甀漀攀礀瘀伀搀嘀栀攀昀娀欀焀瀀刀洀琀洀砀挀栀戀欀瘀一吀最螋谀啓膗蝎摣昀搀昀搀挀愀挀搀戀搀挀昀螋葎豶啓膗蒏蝎堀腤蝎蒎祶豰葧蕶腓捙湫蝎蒋祶諿蝎蒋盿捎湫鑸豵癸扔顣豣豔葛鑙葰豢0猰啓蝎00葎蝶祓0İ蝎蝎当腎鱒尰臿魧葒蝎筟婮祲葲v扎諿葎祶İ葎祶堰葝祶葝祶虮拿葢蝎象0蝎葓罣蝎f祎炘葝魼葒镶敢魎椰
28、魲鮍葒閑敢岀虾W扥罗術葙譶魓挰著譶魓豒蒄翿靏蝎扥奧鮀蝎敞虧N驎葛祺捒棿屝屝慏灎鮏鹎虒蝎蒋搰撖幎蹜魒蒖塎g蟿獬佞乐荟莍蝎葓敭罣敞虧0蝎腓魧葒彾葎鱢苿汙攰腨矿g豠沐葠祶瑎轝佹屔葏蹵祺豠荳葘蒑遶偧葞栰抈蒗裿腎鶉恏葎豟萠腜蝎筟膘魧葒彾塟葏奠葎葎屝葎屝屏衎摭獥萠0蝎蝎蒕鉶屎婹葢菿膗蹨葎魎N蒍vy葎蝎蹛敎荳灘屝N瑎葎蠰屝慏祻筟膘T萠蝎葹蜠魓传貌獧Q陮塦塎煺葔拿魧葒蕶婑鮀豒魟0蝎褠桛蝑葎屝驑虛襎桛蝑蝎筟悘衧葙塏熚楓岖敎腨汨敓槿脰罨屝艳燿楓灼荥繥晧襓桛譑葝覎桛b驎葞虎襙葟屝筟誘襢桛陗盿属蝎蒋襧桛鱏絵鱏屝鱏0蝎蒋0脀遒啓葎貄屝臿豰豔葛葎屝靏豥灙源虺幼İ臿襺蹙蚈筟南葺靎屔孺鮀睻卑豨貚葥蝎蒋鱢盿繧塢著颕睎卑敭葑屝鹏鹷絵魔葒蝎蝎蕓腑睧魧葒偠饧瞏卑葏屝鹏乓貉鹔蒈蝎0脀敺桐蝎筓譼蝎菿婛N蝹觿葎扥抗筟湨葎屝靏屝祏豰屝敺桐蝎葓筶譼WS膗豏蕎葑獶潏敠遧盿腞蒉蹕齎f腥腓艾葧膗钉腛譓葎螖黿汒塎T膗澉綂葙幼批貗葜卶蝟蝒靎屝蟿筓譼彗筎鮏筟蹎葎靶豥0脀魙癸塎葝葎塶葝荫轏罢蒕忿葥瑶筞墏葎罢荳夰屝扱葔赶鈰敎鉢葥彶扜虢葧蝶赎蝢腑煣蝎k塏塏葎霰敥潒玖煗虔塎蝎葓豖摖蝎筟膘冀墆葝0脀豺豔葛蝎蒋蒐豟鉹博睝稰絧厏擿撖钏葎墉葒豺艫著聶晫孺霰椰幼噹葫喀葾屓奐W桟沈噓腦敭顓葎蕶貍硔譑顣董犋虝祹該葫蝎葓鞉響骀0蝎蒋0尰f葧屝問蝎蒋靶罦魙鸰葐忿啦扎慟蒌螀旿遧蹮仿蹨葭鹷饛村葮襶蝎筟誘蝗屓諿魼福牲陥葜蝗葏著獬凿葎潶綂扙慟
