Automatic Extration of Web Data

1
Automatic Extration of Web Data --Web ???????
2
????

• Nora Derouiche, Bogdan Cautis, Talel Abdessalem.
  Automatic Extraction of Structured Web Data with
  Domain Knowledge.IEEE ICDE 2012
• TalelAbdessalem,BogdanCautis,NoraDerouiche.ObjectR
  unner Lightweight, Targeted Extraction and
  Querying of Structured Web Data.
• Bing Liu?. ????.Web ????(?2?).???????
• ???.Web??????????.??????????

3
part1 ???????
part2 ????
part3 ObjectRunner
part4 ????
4
???????
5
???????
????/??(Information Extraction) ?????????????????
?(???????)????(????????)???????????????????
?????8???
??(??)??,??(??)??
6
???????
????
7
???????
??????????????

1. ????????????,?????????(???)
2. ?????????????????????(????)?

8
MUC?IE????

• 5????????
• - NE (Named Entities)
• - ER (Entity Relations)
• - Template Scenario (????)
• - Coreference (??)
• - Template Merger(????)
• ?????? NE, ER, Scenario ?????Coref, Merger ???????

??MUC??Message Understanding Conference
9
5????????

• - NE (Named Entities)????????????,???????/??????
  ?????
• - ER (Entity Relations)?????????????(??)
• Write_of(??,??),
• price_of(??????(???) , ? 47.70  )
• - Template Scenario (Event Structures)????
• ????(Timeltgt, Spotltgt, Convenerltgt,
  Topicltgt)
• - Coreference (Identity descriptions)
  ???????
• - Template Merger ???????

10
???????
????????? 1)????(Manual Approach)??????????,????
???????(Pattern),???????????????? 2)?????(Wrapper
Introduction)????????,???????????????????????????
?????,????????????????????????????? 3)????(Automat
ic Extration)??????????????????,???????????????,?
?????????????????????,?????????????????????
11
???????

• ??????
• ????????????????
• ????????????Web????????,???????????????????????,?
  ??????????????????

?????????????????????,?????????!!
12
???????
Web???????
13
???????
???????????????XHTML/XML????
???????????????????????/???0???????????(?????)????
?
??????????????,???????????????,????????????????
???????????DOM??????,????????????????????,????????
????????????
???????????????????????????
14
???????
?????????
?? ???????????????,????????????? ???????????????
????????,?????????????????????,???????????????????
???????(???????????????)?
15
???????
16
???????
17
????
?? ????????????????(???3?)
??  ??? (??), Micheline Kamber (??), ?? (??), ?? (??), ??? (??) 
??? ???????
???? 2012-08??
??/?? ??
?? ? 60.90 
????
???? ?????????????
??(??) ?????80??2?25?????? 
?????? 57
?????? ? 51.2
18
???????
????
19
???????

• ltHTMLgtltbodygt
• ltdiv id"atfResults" class"list results
  apsList"gtltdiv id"result_0" class"fstRow prod
  celwidget" name"B008RVQHHE"gt ltdiv
  class"linePlaceholder"gtlt/divgtltdiv class"image
  imageContainer"gt lta href"http//www.amazon.cn/E8
  AEA1E7AE97E69CBAE7A791E5ADA6E4B89
  BE4B9A6-E695B0E68DAEE68C96E68E98-E
  6A682E5BFB5E4B88EE68A80E69CAF-E99F
  A9E5AEB6E7829C/dp/B008RVQHHE/refsr_1_1?sb
  ooksampieUTF8ampqid1368498914ampsr1-1amp
  keywordsE695B0E68DAEE68C96E68E98"
  target"_blank"gtltimg src"http//ec4.images-amazon
  .com/images/I/51ydR-eztrL._SL160_PIsitb-sticker-ar
  row-dp,TopRight,12,-18_SH30_OU28_AA160_.jpg"
  class"productImage" alt"??????"/gtlt/agtlt/divgt
• lth3 class"newaps"gt lta href"http//www.amazon.cn
  /E8AEA1E7AE97E69CBAE7A791E5ADA6E4
  B89BE4B9A6-E695B0E68DAEE68C96E68E9
  8-E6A682E5BFB5E4B88EE68A80E69CAF-E
  99FA9E5AEB6E7829C/dp/B008RVQHHE/refsr_1_1
  ?sbooksampieUTF8ampqid1368498914ampsr1-1
  ampkeywordsE695B0E68DAEE68C96E68E98
  " target"_blank"gtltspan class"lrg"gt357453163
  9264263118523398199952007065306
  259682545425366254962701024565
  199822521626415(2140720070315323
  29256)lt/spangtlt/agt ltspan class"med
  reg"gt????Micheline Kamber???? ?? ???????
  (2012-08??)lt/spangt lt/h3gt
• ltligt ltspan class"med avblity"gt ????ltspan
  class'bld grn'gt??lt/spangt?ltspan class'bld
  grn'gt??lt/spangt????lt/spangt lt/ligtltli class"sect
  mbc"gt?????? - ??lt/ligt
• ltli class"med grey mkp2"gt lta href"http//www.am
  azon.cn/gp/offer-listing/B008RVQHHE/refsr_1_1_olp
  ?sbooksampieUTF8ampqid1368498914ampsr1-1
  ampkeywordsE695B0E68DAEE68C96E68E98
  ampconditionnew" target"_blank"gtltspan
  class"price bld"gt?51.20lt/spangt ?? ltspan
  class"grey"gt(57??)lt/spangtlt/agtlt/ligtlt/ulgt
• ltspan class"rvwCnt"gt(lta alt"??4.4 ?"
  class"longReview" href"http//www.amazon.cn/E8
  AEA1E7AE97E69CBAE7A791E5ADA6E4B89B
  E4B9A6-E695B0E68DAEE68C96E68E98-E6
  A682E5BFB5E4B88EE68A80E69CAF-E99F
  A9E5AEB6E7829C/product-reviews/B008RVQHHE/re
  fsr_1_1_cm_cr_acr_txt?ieUTF8ampshowViewpoints
  1" target"_blank"gt30???lt/agt)lt/spangt lt/ligt
• ltligt ?????80??2?25?????? lta href"http//www.amaz
  on.cn/E8AEA1E7AE97E69CBAE7A791E5ADA
  6E4B89BE4B9A6-E695B0E68DAEE68C96E6
  8E98-E6A682E5BFB5E4B88EE68A80E69C
  AF-E99FA9E5AEB6E7829C/dp/B008RVQHHE/refs
  r_1_1_det?sbooksampieUTF8ampqid1368498914a
  mpsr1-1productPromotions" target"_blank"gt?????
  ?lt/agtlt/ligt
• lt/HTML gtlt/bodygt

20
???????
????
21
???????
???????
ltNamedEntitiesgt ltBookListgt ????????????????(??
?3?) ??????(???)  lt/ BookListgt lt/NamedEntitiesgt
ltEntityRelationsgt write_of(??????(???) ?? ,
???(Pang-Ning Tan)?????(Michael
Steinbach)????(Vipin Kumar)? ??) lt/EntityRelations
gt
22
???????
Sample pages
23
???????
Entity type ti Complex types defined in reclusive
manner(??) t ti, mi, tia set of instances
of type ti (atomic or not) mi a multiplicity
constraint that specifies restrictions on the
number of ti instances in t n-m,(0??),(1??),?(0
??),1

• Recognizer ri
• user-defined regular expressions
• system predefined ones (e.g., addresses,dates,
  phone numbers, etc)
• open, dictionary-based ones (isInstanceOf
  recognizers)

Structured Object Description(SOD) SOD???????????,
?????????,???????????
24
???????
Structured Object Description(SOD) SOD???????????,
?????????,???????????
????????SOD s????S,??????SOD s ????Si????????? SOD
s????????type t ti ,mi ,???????sept(???),??
?????ti????????? ??????type t t1,...,tk
,??????k????k??????t,?????????
25
???????
Structured Object Description(SOD)
?????????
26
??????SOD s????S1,,Sn???
1.?s??????(entity types)???????(type
recognizer) 2.?????Si,????? 1)??????????????? 2
)?????????? 3)????????????i(s,Si) 4)?????i????Si
?s????? 3.??????????????
The extraction template
27
???????
28
???????

• ??????????
• ????
• ???????

29
???????
?????

1. ????
2. ??????????(?????)
3. ??JTidy?????HTML?????XML??

A.???????(Type recognizers)
1.?????( YAGO??) 2.?Web??????????????
30
???????
B.?????????

• 1.????(??????k),?Si,SOD s
• 2.???S Si
• 3.?s??????????????
• 4.??s????????t
• 5. ?????S??t
• 6. ?????????
• ??S???????????S,??S S
• 9.???S??????k???(????????)

31
B.?????????
32
???????
??
???????p1????DOM?,?????????DOM???,?????????????DOM
?
33
???????
????
count(i,t,p)--(i,t)????????p???? count(i)?i???????
?? count2525?????
34
???????
????
??,tf(i)term frequency(?????)
35
???????
C.?????(Wrapper)
??SOD s ???????
????(occurrence vector) ??p1,p2,p3??ltdivgt?????????
?lt3,3,6gt ???EQs??????????????
36
???????
?p1?p2?ltdivgt????,??p1 .ltdivgt?p2.ltdivgt???????
37
???????
?????(non-conflicting annotations
) ?token(html??)??????????????????,??????token???
???? ????(conflicting annotations) ?token???????
?,??????token??????
38
???????
C.?????(????HTML????)

• ??????HTML???????
• 1.repeat
• repeat
• ?????
• ???????
• if ?????? then
• ????
• end if
• ??Eqs????????????
• ?? fixphoint
• ??????????????????
• ?? fixphoint

39
???????
C.?????(????HTML????)
40
???????
D.????
???SODs?????????t1,,t3?t31,t32????????t2
41
???????
D.????
42
???????
E.?????????
????? ?????SOD 1.?????????????????,??????????? 2.?
???????(?????,???????50)
???????? ??????,???????(???2)
43
ObjectRunner
44
ObjectRunner system
45
ObjectRunner system
Figure Extraction interface
46
ObjectRunner system
Figure Query interface
47
???????
48
???????
Datasets Five different domains concerts,
albums, books, publications and cars
Concerts
Books
price
Albums
Publications
Cars
Car brand
price
49
???????
????????????????
???????????
50
???????
Correct attributesAc Correct objectsOc Partially
correct attributesAp Partially correct
objectsOp Incorrect attributesAi Incorrect
objectsOi The pecision for correctnessPc The
precision for patially-correctness--Pp
??,NoOcOiOp
51
(No Transcript)
52
???????
Sample selection vs. random selection
?????????ObjectRunner?????,???????????????????????
????????????????,????SOD??????????????????????
53
???????
Comparison with state-of-the-art approaches
?????????ObjectRunner???????RoadRunner,?RoadRunner
?????????????,????????0?
54
???????
55
???????
56
Thank you !(QA)
57
???
58
?W(A,B )??A?B???????????, m(ltA1,,Akgt, ltB1,,Bn
gt)?ltA1,,Akgt? ltB1,,Bn gt??????????? ??RA!RB,??W(A
,B )0??W(A,B )????????
59
?????(STM)
60
?????(STM)
61
(No Transcript)
62
????

• ?????????????????????????????????????????????????
  ?????????????????????????,??????????
• ????(Pattern matching)???????????????????????????
  ??????,?????????
• ????????????????????????????,????????????????????
  ????????????,??,?????????????

63
?????
64
?????