Deep learning, especially deep convolutional neural. networNs (CNNs) [1] ... of such newly developed networNs exerts the pressure of throughput and energy ...
2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture
3UHGLFWLRQEDVHG([HFXWLRQRQ'HHS1HXUDO1HWZRUNV
0LQJFRQJ6RQJ-LHFKHQ=KDR@± >@ 0DQ\ DFFHOHUDWRU GHVLJQV OHYHUDJH SUXQLQJ EDVHG WHFKQLTXH >@ >@ WR VKULQN WKH VL]H RI RULJLQDO GHQVH QHWZRUNV$OWKRXJKSUXQLQJFDQUHGXFHWKHFRPSXWDWLRQDODQG PHPRU\RYHUKHDGVLWUHTXLUHVUHWUDLQLQJWKHSUXQHGQHWZRUN WR UHVWRUH WKH DFFXUDF\ 7KH WUDLQLQJ SURFHGXUH LV WLPH FRQVXPLQJDQGWKXVORVHVWKHIOH[LELOLW\)RULQVWDQFHLWWDNHV WKUHHZHHNVWRWUDLQ9**1HWRQIRXUKLJKHQG19,',$7LWDQ *38V 2WKHU ZRUNV GHYRWH WR UHGXFLQJ WKH FRPSXWDWLRQDO RSHUDWLRQVYLDDSSUR[LPDWHFRPSXWLQJEDVHG PHWKRGV>@± >@1HYHUWKHOHVVWKHLUSHUIRUPDQFHLPSURYHPHQWFRPHVDW WKHH[SHQVHRIVRPHORVVLQQHWZRUNDFFXUDF\7KHUHIRUHLWLV GHVLUDEOH WR H[SORUH WHFKQLTXHV WKDW FDQ HIIHFWLYHO\ UHGXFH FRPSXWDWLRQDQGPHPRU\RYHUKHDGVZKLOHDYRLGLQJUHWUDLQLQJ DQGDFFXUDF\ORVV $ SURPLVLQJ WUHQG LV WR UHPRYH WKH XQQHFHVVDU\ FDOFXODWLRQVDFURVVWKHQHXURQQHWZRUNVDQGWKXVVDYLQJWKH WLPHDQGHQHUJ\$QLQLWLDOSURSRVDOQDPHGCnvlutin>@DLPV DWHOLPLQDWLQJ]HURYDOXHGRSHUDQGPXOWLSOLFDWLRQVWRUHGXFH FRPSXWDWLRQDORSHUDWLRQVZLWKRXWWKHVDFULILFHRIDFFXUDF\,W LVLQVSLUHGE\WKHIDFWWKDW5HFWLILHU/LQHDU8QLW 5H/8 UHVXOWV LQDEXQGDQW]HURVLQWKHLQSXWIHDWXUHPDSV>@ +RZHYHU ZH REVHUYH WKDW WKH ]HURUHPRYDO WHFKQLTXH DGRSWHGE\CnvlutinVWLOOQRWIXOO\H[SORLWVWKHRSSRUWXQLWLHV LQ '11V )LUVW PDQ\ RXWSXW QHXURQV DIWHU QRQOLQHDULW\ OD\HUV VXFK DV 5H/8 DQG 0D[SRROLQJ DUH VWLOO LQHIIHFWXDO HYHQLIWKH]HURUHPRYDOKDVEHHQDSSOLHG6LQFHDQLQHIIHFWXDO RXWSXWQHXURQFDQQRWSDVVLWVYDOXHWRQH[WOD\HUWKLVPHDQV DOO WKH FRPSXWDWLRQV LQFOXGLQJ QRQ]HURYDOXHG RSHUDQG PXOWLSOLFDWLRQV UHODWHGWRWKHVHRXWSXWQHXURQVDUHIXWLOHDQG ZDVWHIXO6HFRQGCnvlutinPDLQO\IRFXVHVRQFRQYROXWLRQDO &219 OD\HUV+RZHYHUVLQFHPRVWRIWKHILOWHUZHLJKWVDUH LQ IXOO\FRQQHFWHG )&1 OD\HUV Cnvlutin FRXOG QRW UHGXFH WKH PHPRU\ DFFHVV RYHUKHDG 7KLUG WKH H[LVWHQFH RI 0D[ SRROLQJ 0D[3 OD\HULQ'11VJUHDWO\UHGXFHVWKHDPRXQWRI ]HURV DQG WKLV FDQ VWXOWLI\ WKH ]HURUHPRYDO WHFKQLTXH RI Cnvlutin ,QVWHDG RI VLPSO\ HOLPLQDWLQJ WKH LQHIIHFWXDO RSHUDQG PXOWLSOLFDWLRQVZKHUHRQHRIWKHLQSXWVLV]HURZHRSWWRVNLS RYHU WKH FRPSOHWH FRPSXWDWLRQ SURFHVV WKDW UHODWHV WR LQHIIHFWXDORXWSXWQHXURQV,QWKLVSDSHUZHGHILQHLQHIIHFWXDO RXWSXWQHXURQV L(21 DVWKRVHRXWSXWQHXURQVWKDWKDYHQR
,QHIIHFWXDO0XO5DWLR
6NLS,QSXW=HURV
6NLSL(21VZLWK3UHGLFWLRQ
2YHUDOO
D $OH[1HW
E 9**
2YHUDOO
)LJXUH&RPSDULVRQRI,QHIIHFWXDO0XOWLSOLFDWLRQV
LQIOXHQFHRQWKHVXEVHTXHQWOD\HUVVLQFHWKH\DUHILOWHUHGRXW E\WKHQRQOLQHDULW\OD\HUVLQ'11VVXFKDV5H/8DQG0D[3 )RU 5H/8 RQO\ WKH RXWSXW QHXURQV ZLWK SRVLWLYH YDOXH FDQ SDVVWRWKHQH[WOD\HU)RU0D[3RQO\WKHRXWSXWQHXURQZLWK WKHPD[LPXPYDOXHLQDVXEUHJLRQFDQSDVVWRWKHQH[WOD\HU ,IZHFDQSUHGLFWWKHL(21VLQDGYDQFHZHFDQVNLSRYHUWKH ZKROHFRPSXWDWLRQSURFHVV LQFOXGLQJWKRVHQRQ]HURYDOXHG RSHUDQG PXOWLSOLFDWLRQV UHODWHV WR WKHVH L(21V DQG JUHDWO\ LPSURYHSHUIRUPDQFH &RPSDUHG ZLWK UHPRYLQJ LQHIIHFWXDO PXOWLSOLFDWLRQV UHODWHV WR ]HUR LQSXWV FRPSOHWHO\ VNLSSLQJ RYHU L(21V PDQLIHVWVWKHIROORZLQJDGYDQWDJHV LWRIIHUVPRUHEHQHILWV RQ SHUIRUPDQFH DQG HQHUJ\ VLQFH VNLSSLQJ RYHU L(21V FDQ HOLPLQDWHPRUHPXOWLSOLFDWLRQVLQHDFKOD\HUDVVKRZQLQ)LJ LWFDQDOVRWDFNOHWKHFKDOOHQJHRIPHPRU\DFFHVVVLQFH HDFKRXWSXWQHXURQKDVLWVRZQILOWHULQ)&1OD\HUVDQGWKHUH LVQRQHHGWRORDGWKHILOWHUVZKHQWKHLUFRUUHVSRQGLQJRXWSXW QHXURQVDUHLQHIIHFWXDO DOWKRXJK0D[3LVXQIDYRUDEOHIRU Cnvlutin LW LV D JRRG RSSRUWXQLW\ IRU XV WR H[SORUH VLQFH LW SURGXFHV PRUH L(21V DQG LW FDQ EH VWDFNHG ZLWK RWKHU LQHIIHFWXDOQHXURQUHPRYDOWHFKQLTXHVVXFKDVCnvlutin)RU WKRVH SUHGLFWHG HIIHFWXDO RXWSXW QHXURQV (21 ZH FDQ IXUWKHU UHGXFH LWV ]HURRSHUDQG PXOWLSOLFDWLRQV WR DFKLHYH KLJKHUVSHHGXS 7R SUDFWLFDOO\ VNLS RYHU WKH L(21V ZH SURSRVH WR WUDQVIRUP WKH FRPSXWLQJ SDWWHUQ RI '11V IURP RQHWLPH H[HFXWLRQ WR D WZRVWDJH SUHGLFWLRQEDVHG H[HFXWLRQ )LUVW WKHSUHGLFWRUSUHGLFWVWKH(21V 3UHGLFWLRQ6WDJH 7KHQWKH H[HFXWRURQO\QHHGVWRSHUIRUPFRPSXWDWLRQVUHODWHGWR(21V ([HFXWLRQ 6WDJH )XUWKHU ZH PD[LPL]H WKH FRPSXWDWLRQDO UHXVHEHWZHHQWKHVHWZRVWDJHVWRDPRUWL]HWKHFRPSXWDWLRQDO RYHUKHDG LQWURGXFHG E\ 3UHGLFWLRQ 6WDJH 6SHFLILFDOO\ WKH UHVXOWVRIWKHSUHGLFWRUDUHQRWRQO\XVHGIRUSUHGLFWLRQEXWDOVR FDQEHXWLOL]HGE\WKHH[HFXWRU7KXVWKHH[HFXWRURQO\QHHGV WRSHUIRUPWKHUHPDLQLQJFDOFXODWLRQRI(21VDQGREWDLQWKH ILQDO UHVXOW E\ DGGLQJ WKH SDUWLDO UHVXOW IURP SUHGLFWRU FRQVHTXHQWO\ DFKLHYLQJ QHDULGHDO VSHHGXS DQG QR DFFXUDF\ ORVV0RUHRYHUZHGHVLJQDXQLIRUPVHULDOSURFHVVLQJHOHPHQW USPE IRUERWKSUHGLFWRUDQGH[HFXWRUWRJHWULGRIWKHH[WUD DUHDRYHUKHDG 7R PD[LPL]H WKH SURFHVVLQJ WKURXJKSXW ZH IXUWKHU SURSRVH D VFDOHRXW GHVLJQ WR SURFHVV WKH PXOWLSOH RXWSXW QHXURQV LQ SDUDOOHO +RZHYHU WKH VSDUVLW\ RI (21V LQ WKH RXWSXW IHDWXUH PDSV FKDOOHQJHV WKH VFDOHRXW GHVLJQ DQG UHVXOWVLQWKHLGOHQHVVRIUSPE:HOHYHUDJHSUHIHWFKORDGLQJ DQG RXWRIRUGHU H[HFXWLRQ WR PDNH USPEV EHLQJ IXOO\
XWLOL]HG :H DOVR SURSRVH DQDO\WLFDO PRGHOV WR GHPRQVWUDWH RXUPHPRU\GHVLJQWKDWFDQSURYLGHHQRXJKGDWDIRUUSPEV (YDOXDWLRQ UHVXOWV RYHU D VHW RI VWDWHRIWKHDUW '11V VKRZ WKDW RXU SURSRVHG GHVLJQ DFKLHYHV DQ DYHUDJH ; VSHHGXS DQG ; HQHUJ\HIILFLHQF\ RYHU WKH WUDGLWLRQDO DFFHOHUDWRU2XUSUHGLFWLRQEDVHGGHVLJQKDVDQDYHUDJH; VSHHGXS RYHU Cnvlutin >@ DQG ; VSHHGXS RYHU Stripes >@ 0RUHRYHU E\ FRPELQLQJ RXU GHVLJQ ZH FDQ LPSURYH Cnvlutin DQG Stripes E\ ; DQG ; RQ DYHUDJH UHVSHFWLYHO\ ,QVXPPDU\ZHPDNHWKHIROORZLQJNH\FRQWULEXWLRQV x:H REVHUYH DQ RSSRUWXQLW\ WR VLJQLILFDQWO\ LPSURYH WKH SHUIRUPDQFH DQG HIILFLHQF\ RI '11V E\ FRPSOHWHO\ E\SDVVLQJWKHFRPSXWDWLRQRIL(21VEURXJKWE\WKHQRQ OLQHDULW\OD\HUVVXFKDV5H/8DQG0D[3 x:HSURSRVHDWZRVWDJHSUHGLFWLRQEDVHG'11H[HFXWLRQ PRGHO ZLWKRXW VDFULILFLQJ WKH DFFXUDF\ 7KLV PRGHO HPSOR\V3UHGLFWLRQ6WDJHWRLGHQWLI\WKH(21VIRU&219 DQG )&1 OD\HUV DQG UHXVHV WKH SUHGLFWLRQ UHVXOWV DV LQWHUPHGLDWH UHVXOWV WR LQFUHPHQWDOO\ FRQGXFW WKH H[HFXWLRQRQ(21V x:HSUHVHQWDXQLIRUP VHULDOSURFHVVLQJHOHPHQW USPE IRUERWKSUHGLFWRUDQGH[HFXWRUWRLPSURYHWKHIOH[LELOLW\ DQGPLQLPL]HWKHDUHDRYHUKHDG x:HSURSRVHWKHVFDOHRXWGHVLJQRIUSPEWRPD[LPL]HWKH SURFHVVLQJWKURXJKSXW 7KHUHVWRIWKLVSDSHULVRUJDQL]HGDVIROORZV6HFWLRQ,, LQWURGXFHV WKH EDFNJURXQG DQG PRWLYDWLRQ 6HFWLRQ ,,, LOOXVWUDWHV RXU SUHGLFWLRQEDVHG PLFURDUFKLWHFWXUH GHVLJQ 6HFWLRQ ,9 GHVFULEHV WKH FKDOOHQJHV DQG RSWLPL]DWLRQ LQ WKH VFDOHRXW GHVLJQ 6HFWLRQ 9 HYDOXDWHV RXU GHVLJQ 5HODWHG ZRUNVDQGFRQFOXVLRQVDUHGLVFXVVHGLQ6HFWLRQV9,DQG9,, UHVSHFWLYHO\ ,,
%$&.*5281'$1'027,9$7,21
A. Non-linearity and Computation Overheads in DNNs 'HHS OHDUQLQJ WHFKQLTXHV HPSOR\ PXOWLSOH QHXUDO QHWZRUN OD\HUV WR OHDUQ OHYHOV RI UHSUHVHQWDWLRQ DQG DEVWUDFWLRQWKDWPDNHVHQVHRIGDWDVXFKDVLPDJHVRXQGDQG WH[W &RQYROXWLRQDO &219 DQG )XOO\ FRQQHFWHG )&1 OD\HUV DUH WKH WZR PRVW LPSRUWDQW GHHS OHDUQLQJ QHWZRUN OD\HUV $OWKRXJK &219 DQG )&1 DFFRXQW IRU PRVW RI WKH FRPSXWDWLRQVLQ'11VDVOLQHDUIXQFWLRQVWKH\RQO\IRUPD QHZOLQHDUFRPELQDWLRQDQGGRQRWFUHDWHTXDOLWDWLYHO\QHZ
753
,QSXW;
)HDWXUH0DSV #;
)0DSV
)0DSV
)0DSV
)0DSV
#;
#;
#;
#;
KƵƚƉƵƚ&ĞĂƚƵƌĞDĂƉ
ϰϬϵϲ ϰϬϵϲ
^sD
KEsZĞ>h
KEsZĞ>h DĂdžWŽŽůŝŶŐ
KEsZĞ>h DĂdžWŽŽůŝŶŐ
KEsZĞ>h KEsZĞ>h DĂdžWŽŽůŝŶŐ
)HDWXUH([WUDFWRU
͍ ůĂƐƐ
/ŶƉƵƚ&ĞĂƚƵƌĞDĂƉƐ
Ͳϭ͘Ϯ
Ϭ͘ϴ
ͲϬ͘ϰ
ͲϬ͘ϱ
Ϭ͘Ϯ
ϭ͘ϭ
Ϭ͘ϲ
ϭ͘Ϯ
ͲϬ͘ϵ
ͲϬ͘ϳ
ͲϬ͘ϱ
ͲϬ͘ϯ
Ϭ͘Ϯ
ͲϬ͘ϴ
ZĞ>h
Ϭ͘ϯ
Ϭ
Ϭ
Ϭ͘ϴ
Ϭ
Ϭ
Ϭ͘Ϯ
ϭ͘ϭ
Ϭ͘ϲ
ϭ͘Ϯ
Ϭ
Ϭ
Ϭ
Ϭ
Ϭ͘Ϯ
Ϭ
Ϭ
Ϭ
Ϭ
Ϭ
Ϭ
Ϭ
ϭ͘ϭ
Ϭ
ϭ͘Ϯ
Ϭ
Ϭ
Ϭ
Ϭ
Ϭ͘Ϯ
Ϭ
DĂdžW Ϭ͘ϯ ;ϮdžϮͿ
;ĂͿ
&ODVVLILHU хϬ
ᬅ
LQIRUPDWLRQ 7R PDNH '11V OHDUQ WKH FRPSOH[ IXQFWLRQDO PDSSLQJ EHWZHHQ LQSXWV DQG RXWSXWV QRQOLQHDU DFWLYDWLRQ IXQFWLRQV DOZD\V IROORZ HDFK &219 DQG )&1 OD\HU 5HFWLILHGOLQHDUXQLW 5H/8 LVWKH PRVWSRSXODUQRQOLQHDU DFWLYDWLRQIXQFWLRQLQWKHVWDWHRIWKHDUW'11V>@VXFKDV 5HV1HW>@*RRJOH1HW>@DQG9**1HW>@,WLQWURGXFHV QRQOLQHDULW\E\RQO\DOORZLQJSRVLWLYHYDOXHVSDVVWKURXJK DQGFRQYHUWLQJDQ\QHJDWLYHLQSXWWR]HUR$QRWKHULPSRUWDQW WHFKQLTXHWKDWLQWURGXFHVQRQOLQHDULW\LVSRROLQJ>@ZKLFK LV D IRUP RI QRQOLQHDU GRZQVDPSOLQJ 7KHUH DUH VHYHUDO QRQOLQHDU IXQFWLRQV WR LPSOHPHQW SRROLQJ DPRQJ ZKLFK PD[SRROLQJ 0D[3 LVWKHPRVWFRPPRQFKRLFH,WSDUWLWLRQV WKHLQSXWLPDJHLQWRDVHWRIUHFWDQJOHVDQGIRUHDFKVXFKVXE UHJLRQRXWSXWVWKHPD[LPXPSL[HO :H LQWURGXFH SUHOLPLQDULHV RI QHXUDO QHWZRUN DUFKLWHFWXUHXVLQJ$OH[1HW>@DVDQH[DPSOH$VVKRZQLQ )LJ LW PDLQO\ FRQVLVWV RI ILYH FRQYROXWLRQDO OD\HUV WZR IXOO\ FRQQHFWHG OD\HUV VHYHQ 5H/8V DQG WKUHH 0D[3V &RQYROXWLRQDO &219 OD\HUVH[WUDFW³IHDWXUHV´E\DGRSWLQJ D FRQYROXWLRQDO NHUQHO RQ LWV LQSXW IHDWXUH PDSV 7KH FRPSXWDWLRQDOSDWWHUQLVDFRQYROXWLRQDORSHUDWLRQFDOFXODWHG DORQJWKUHHGLPHQVLRQVDVVKRZQLQ
)LOWHU:HLJKW
ͲϬ͘ϵ
&E &E ZĞ>h ZĞ>h
)LJXUH$Q2YHUYLHZRI&110RGHO$OH[1HW
2XWSXW1HXURQ
Ϭ͘ϯ
KEs
,QSXW1HXURQ
/ŶƉƵƚ&ĞĂƚƵƌĞDĂƉƐ
ƉƉƌŽdžŝŵĂƚŝŽŶ ĂƐĞĚWƌĞĚŝĐƚŽƌ
Ϭ
Ϭ
хϬ
хϬ
хϬ
Ϭ
Ϭ
хϬ
хϬ
Ϭ
Ϭ
Ϭ
Ϭ
хϬ
Ϭ
>ŽĐĂƚŝŽŶŝŶĨŽƌŵĂƚŝŽŶ
ᬆ
džĞĐƵƚŽƌ ;&ƵůůͲƉƌĞĐŝƐŝŽŶͿ ĨĨĞĐƚƵĂůŽƵƚƉƵƚŶĞƵƌŽŶ /ŶĞĨĨĞĐƚƵĂůŽƵƚƉƵƚŶĞƵƌŽŶ
Ϭ͘ϯ Ϭ͘ϲ
KƵƚƉƵƚ&ĞĂƚƵƌĞDĂƉ
Ϭ͘ϴ Ϭ͘Ϯ ϭ͘ϭ
Ϭ͘ϯ
Ϭ
Ϭ
Ϭ
Ϭ
Ϭ͘Ϯ ϭ͘ϭ
Ϭ͘Ϯ
Ϭ͘ϲ
ϭ͘Ϯ Ϭ
Ϭ
Ϭ
ϭ͘Ϯ
ĂůĐƵůĂƚŝŽŶŽĨ KE
Ϭ
)LJXUH$1DLYH'HVLJQ
Ϭ͘Ϯ
Ϭ͘ϴ Ϭ Ϭ
;ďͿ
WKH(21VRI5H/8DUHWKHQHXURQVZLWKSRVLWLYHYDOXHV7KH (21VRI0D[3DUHWKHPD[LPXPQHXURQVLQDVXEUHJLRQ ,I '11 DFFHOHUDWRUV FDQ SUHGLFW WKH (21V LQ DGYDQFH DQGVNLSWKHFDOFXODWLRQRIL(21VWKHLUSHUIRUPDQFHZLOOEH JUHDWO\ LPSURYHG $V VKRZQ LQ )LJ IRU $OH[1HW WKH RYHUDOOLQHIIHFWXDOPXOWLSOLFDWLRQVUHODWHGWRL(21VDFFRXQWV IRUDURXQGRIWRWDOPXOWLSOLFDWLRQVZKLFKFDQUHVXOWLQ DERXW;VSHHGXS C. Leveraging Predictability: A Naive Design $Q LQWXLWLYH DSSURDFK WR DFKLHYH WKH DERYHPHQWLRQHG RSWLPL]DWLRQLVWRGHVLJQDGHGLFDWHGSUHGLFWRUIRUWKHH[LVWLQJ '11DFFHOHUDWRU:HVKRZDFRQFHSWXDOGHVLJQLQ)LJ E 7KH DSSUR[LPDWLRQEDVHG SUHGLFWRU ILUVW SUHGLFWV WKH ORFDWLRQVRI(21V7KHQWKHH[HFXWRURQO\QHHGVWRIRFXVRQ WKH FDOFXODWLRQ RI (21V )RU WKH LQHIIHFWXDO QHXURQV WKH H[HFXWRUFDQVLPSO\DVVLJQ]HURVWRWKHPVLQFHWKH\KDYHQR LQIOXHQFHRQWKHVXEVHTXHQWOD\HUV'XULQJWKHSUHGLFWLRQZH DUH QRW LQWHUHVWHG LQ WKH VSHFLILF QXPHULFDO YDOXH RI HDFK RXWSXW QHXURQ DQG RQO\ FRQFHUQ DERXW ZKHWKHU WKH\ DUH SRVLWLYH &2195H/8 RU ZKDW WKHLU UHODWLYH YDOXHV DUH &2195H/80D[3 7KHUHIRUH WKHUH LV QR QHHG WR SHUIRUP WKH IXOOSUHFLVLRQ FDOFXODWLRQ LQ WKH SUHGLFWLRQ ,QVWHDG ZH FDQ OHYHUDJH DSSUR[LPDWLRQEDVHG PHWKRG WR GHVLJQWKHSUHGLFWRU &KDOOHQJHV 7KRXJKWKHFRQFHSWXDOGHVLJQFDQVNLSL(21V WKHUHDUHVWLOOVRPHGHVLJQFKDOOHQJHVWKDWOLPLWLWWRDFKLHYH LGHDO VSHHGXS )LUVW WKH ORFDWLRQ LQIRUPDWLRQ RI (21V REWDLQHG IURP WKH SUHGLFWRU FRXOG QRW EH OHYHUDJHG E\ WKH H[HFXWRU LQ 6WHS 7KH H[HFXWRU VWLOO QHHGV WR SHUIRUP WKH IXOOSUHFLVLRQFDOFXODWLRQLQ)LJ E 6HFRQGWKHLQWHJUDWLRQ RISUHGLFWRULQWURGXFHVDGGLWLRQDODUHDRYHUKHDG)LQDOO\WKH VSDUVLW\RI(21VLQWKHRXWSXWIHDWXUHPDSVFKDOOHQJHVWKH WUDGLWLRQDO'11DFFHOHUDWRUSDUDOOHOL]DWLRQGHVLJQ$V'11V FRPHV ZLWK LQWHQVLYH FRPSXWDWLRQDO ZRUNORDGV D W\SLFDO VROXWLRQ LV WR SHUIRUP PXOWLSOH RXWSXW QHXURQV LQ SDUDOOHO VKRZQ LQ )LJ D 6LQFH LW VLPXOWDQHRXVO\ SURFHVVHV PXOWLSOHLQSXWIHDWXUHPDSV ,)0V DQGRXWSXWIHDWXUHPDSV 2)0V ZHQDPHWKLVDUFKLWHFWXUHGHVLJQDV0,)002)0 ZKLFK LV ZLGHO\ XVHG LQ PDQ\ DFFHOHUDWRU >@ >@±>@ VXFKDV'LDQ1DR>@DQG'D'LDQ1DR>@$VVKRZQLQ)LJ D DOO WKH SURFHVVLQJ HOHPHQWV 3(V LQ 0,)002)0 VKDUH WKH VDPH LQSXW ZLQGRZ ,ိ: +RZHYHU GXH WR WKLV
ZKHUH1LVWKHQXPEHURILQSXWIHDWXUHPDSV6LVWKHVWULGH VL]H.\ൈ.[LVWKHILOWHUNHUQHOVL]H1RWHWKDWFDOFXODWLQJRQH RXWSXW QHXURQ LQYROYHV .\ ൈ .[ ൈ 1 PXOWLSO\DFFXPXODWH 0$& RSHUDWLRQV 7KXV &219 LV FRPSXWDWLRQDOO\ LQWHQVLYH)XOO\FRQQHFWHG )&1 OD\HUVDUHXVHGWRPDNHWKH ILQDOLQIHUHQFH7KH\WDNH³IHDWXUHV´LQWKHIRUPRIDYHFWRU IURPDSULRUIHDWXUHH[WUDFWLRQOD\HUPXOWLSO\LWZLWKDZHLJKW PDWUL[ DQG RXWSXW D QHZ IHDWXUH YHFWRU 7KH RSHUDWLRQ LQ )&1FDQDOVREHGHVFULEHGE\ ZKHUH.[DQG.\DUHHTXDO WRDQG1LVWKHOHQJWKRIHDFKNHUQHO,Q)&1OD\HUHDFK RXWSXW QHXURQ LV FRQQHFWHG WR DOO WKH LQSXW QHXURQV LQ WKH SUHYLRXV OD\HU 7KLV UHVXOWV LQ D ODUJH QXPEHU RI NHUQHO ZHLJKWVLQ)&1OD\HUVDQGPDNHVWKHPPHPRU\LQWHQVLYH B. Predicting Output Sparsity: An Opportunity for DNNs $V VKRZQ LQ )LJ WKH EDVLF QHWZRUN DUFKLWHFWXUH FRPELQDWLRQV DUH &2195H/8 &2195H/80D[3 DQG )&15H/8 :H XVH &2195H/80D[3 WR H[HPSOLI\ WKH FRPSXWLQJSURFHVVLQ'11V $VVKRZQLQ)LJ D WKHUHH[LVWPDQ\QHJDWLYHYDOXHV LQWKHRXWSXWIHDWXUHPDSV6LQFH5H/8FRQYHUWVDQHJDWLYH YDOXHWR]HURWKHVHQHJDWLYHRXWSXWQHXURQVFRXOGQRWSDVVWR WKHQH[WOD\HUV$IWHU0D[3OD\HUHYHQVRPHSRVLWLYHRXWSXW QHXURQVFRXOGQRWSDVVWRWKHIROORZLQJOD\HUVEHFDXVH0D[3 OD\HURQO\VHOHFWVWKHPD[LPXPYDOXHLQDVXEUHJLRQ7KXV
754
/>Ɛ
/ͼt
W/
ϭ ϭ ͲϮ Ϭ &ŝůƚĞƌϬ
&ŝůƚĞƌϬ
W Ɛ K&DϬ
&ŝůƚĞƌϭ
W Ɛ K&Dϭ
ಃ
ಃ
&ŝůƚĞƌWŽ
W Ɛ K&DWŽ
;ĂͿ^ĐĂůĞͲŽƵƚ ĞƐŝŐŶ;D/&DͲDK&DͿ
/ͼtϭ Ϯ ϭ Ϭ ϯ
ϭ ϭ ϭ ͲϮ &ŝůƚĞƌϭ Ϯ ϭ Ͳϭ Ͳϭ &ŝůƚĞƌWŽ
Ͳϭ /ĚůĞ
^ĞƌŝĂůDƵůƚŝƉůŝĞƌ
)LJXUH3DUDOOHODQG6HULDO0XOWLSOLHUV /ͼtϮ Ϯ Ϭ Ϯ ϭ
/ͼtϯ Ϭ ϯ ϭ ϯ
ϯ
ͲϮ /ĚůĞ
ϭ
ϱ
Ͳϯ /ĚůĞ
Ϯ
ͲϮ /ĚůĞ
ಃ
ಃ
ಃ
ಃ
Ͳϭ /ĚůĞ
Ϯ
Ϯ
ϭ
LJĐůĞϬͲϯ
LJĐůĞϰͲϳ
LJĐůĞϴͲϭϭ
LJĐůĞϭϮͲϭϱ
;ďͿ/ĚůĞŶĞƐƐŽĨW ƐŝŶD/&DͲDK&D
)LJXUH,GOHQHVVRI3(VLQ6FDOHRXW'HVLJQ
VKDULQJ0,)002)0FRXOGQRWVNLSRYHUWKHL(21VZLWK QHJDWLYHYDOXHVLQ)LJ E ZKLFKUHVXOWVLQWKHLGOHQHVVRI 3(VDQGRIIVHWVWKHSHUIRUPDQFHLPSURYHPHQWEURXJKWE\WKH SUHGLFWLRQ ,,, 35(',&7,21%$6('0,&52$5&+,7(&785('(6,*1 $V GHSLFWHG LQ 6HFWLRQ ,, WKH SUHGLFWDELOLW\ RI QRQ OLQHDULW\ 5H/8 DQG 0D[3 FRXOG EH OHYHUDJHG WR VLJQLILFDQWO\UHGXFHWKHXQQHFHVVDU\FRPSXWDWLRQLQOLQHDULW\ OD\HUV &219DQG)&1 7RPD[LPL]HWKLVEHQHILWZHQHHG WR PLQLPL]H WKH SHUIRUPDQFH RIIVHW LQWURGXFHG E\ WKH SUHGLFWLRQ 7RZDUGV WKLV JRDO ZH SURSRVH D WZRVWDJH FRPSXWLQJ SDWWHUQ IRU WKH OLQHDULW\ OD\HUV 7KH FRPSXWLQJ SDWWHUQ PD[LPL]HV WKH FRPSXWDWLRQ UHXVH EHWZHHQ WKH H[HFXWRU DQG WKH SUHGLFWRU 6SHFLILFDOO\ WKH H[HFXWRU RQO\ QHHGVWRSHUIRUPWKHUHPDLQLQJFDOFXODWLRQRI(21VEDVHGRQ WKH SUHGLFWLRQ UHVXOWV ZKLFK UHGXFHV LWV FRPSXWDWLRQDO RYHUKHDG 0RUHRYHU ZH SURSRVH D XQLIRUP DUFKLWHFWXUH USPEIRUERWKSUHGLFWRUDQGH[HFXWRUWRJHWULGRIWKHH[WUD DUHD RYHUKHDG DQG JDLQ IOH[LEOH FRQILJXUDELOLW\ IRU FRPSXWDWLRQLQOLQHDULW\OD\HUV A. Prediction Rationale (Computation Reuse) :H ILUVW LQWURGXFH WKH UDWLRQDOH RI SUHGLFWLRQEDVHG H[HFXWLRQPRGHOIRUFRPSXWDWLRQUHGXFWLRQ$FFRUGLQJWR WKHFDOFXODWLRQRIRQHRXWSXWQHXURQ 2 FDQEHVLPSOLILHGDV WKH VXP RI SURGXFWV RI LQSXW QHXURQV , DQG ILOWHU ZHLJKWV : $VWKHLQSXWQHXURQV , FDQEHVHSDUDWHGLQWRKLJKRUGHU ELWV IHBs DQG ORZRUGHUELWV ILBs WKHFDOFXODWLRQLQ LV HTXDO WR WKH VXP RI WZR SDUWV VKRZQ LQ WKH VXP RI SURGXFWVRIIHBsDQG:DQGWKHVXPRISURGXFWVRIILBsDQG: ܱ ൌ ܹ ൈ ܫൌ ܹ ൈ ܫு௦ ܰ ا௦ ܫ௦ ൌ ሺ ܹ ൈ ܫு௦ ሻ ܰ ا௦ ܹ ൈ ܫ௦
3UHGLFWLRQ6WDJH
([HFXWLRQ6WDJH
н фф
t
WĂƌĂůůĞů DƵůƚŝƉůŝĞƌ
)LJXUH7ZR6WDJH$UFKLWHFWXUH /ͼtϬ ϭ Ϯ Ϯ Ϭ
>^
>^
džĞĐƵƚŝŽŶ^ƚĂŐĞ
н фф
755
WƌĞĚŝĐƚŝŽŶ Θ н džĞĐƵƚŝŽŶ н
н фф
&RP
DyͺKƵƚ DyͺdĂďůĞ
6LJQ
ZĞ>hͺdĂďůĞ
^ƚĂŐĞ
н 6KLIWHU
н
hŶŝĨŽƌŵƌĐŚŝƚĞĐƚƵƌĞ
WƌĞĚŝĐƚŝŽŶ^ƚĂŐĞ
/,Ɛ />Ɛ
фф
н фф
Ă
KƵƚ Ϭ
&WUO
^ƚĂŐĞ
н
D^
Ă
6KLIWHU
3UHGLFWRU
>^
Ă
t
y
D^
([HFXWRU
D^
WƌĞĚŝĐƚŝŽŶ
>^
н
>^ 3WR6
Ă
н
Ă
y y
D^
Ă
t
н
/,Ɛ
D^
н
Ă
Wů
y y
KƵƚ
KƵƚ Ϭ
)LJXUH8QLIRUP6HULDO3( USPE
ZKHUH NLBs LQGLFDWHV WKH QXPEHU RI ELWV LQ ILBs DQG UHSUHVHQWVWKHVKLIWLQJRSHUDWLRQ7KHQWKHFDOFXODWLRQRIRQH RXWSXWQHXURQFDQEHEURNHQGRZQLQWRWZRVWDJHVDVVKRZQ LQ 6LQFH WKH YDOXH RI WKH RXWSXW QHXURQ LV PDLQO\ GRPLQDWHG E\ WKH FDOFXODWLRQ UHODWHG WR KLJKRUGHU ELWV 3UHGLFWLRQ6WDJH WKHFDOFXODWLRQUHVXOWVRI3UHGLFWLRQ6WDJH FDQEHOHYHUDJHGWRSUHGLFWWKHSRVLWLYHRUQHJDWLYHVLJQRI RXWSXW QHXURQV &2195H/8 RU WKHLU UHODWLYH YDOXHV &2195H/80D[3 7KHUHIRUH DV VKRZQ LQ )LJ WKH SUHGLFWRULVUHVSRQVLEOHIRUWKHFDOFXODWLRQRI3UHGLFWLRQ6WDJH DQGWKHUHPDLQLQJFDOFXODWLRQ ([HFXWLRQ6WDJH LVDVVLJQHG WRWKHH[HFXWRU)LUVWWKHSUHGLFWRUSUHGLFWVWKHORFDWLRQVRI (21VRQO\XVLQJKLJKRUGHUELWVRILQSXWQHXURQV7KHQIRU HDFK RI WKHVH (21V WKH H[HFXWRU FRQWLQXHV WR FRQGXFW WKH UHPDLQLQJ FDOFXODWLRQ ZLWK WKH ORZRUGHU ELWV RI LQSXW QHXURQV )LQDOO\ WKH H[HFXWRU REWDLQV WKH ILQDO YDOXHV RI (21V E\ DGGLQJ LWV UHVXOWV ZLWK WKH UHVXOWV RI 3UHGLFWLRQ 6WDJH UHXVLQJWKHUHVXOWVRISUHGLFWRU 5HFDOO WKH GLVFXVVLRQ LQ 6HFWLRQ ,,$ FRPSXWLQJ RQH RXWSXW QHXURQ LQYROYHV .\ ൈ .[ ൈ 1 PXOWLSO\DFFXPXODWH 0$& RSHUDWLRQV ZKLFK LV FRPSXWHLQWHQVLYH 7R DFFHOHUDWH'11H[HFXWLRQDVVKRZQLQ)LJZHHTXLSWKH SUHGLFWRUDQGH[HFXWRUZLWKPXOWLSOHPXOWLSOLHUVVRWKDWWKH\ FDQSURFHVVPXOWLSOH0$&RSHUDWLRQVLQSDUDOOHO6SHFLILFDOO\ ZH XQUROO LWV LQSXW IHDWXUH PDSV DQG WKH SDUDOOHOLVP JUDQXODULW\ LV 3, ZKLFK DFFRXQWV IRU WKH QXPEHU RI PXOWLSOLHUV 7RUHGXFHWKHRYHUKHDGRISUHGLFWLRQZHVKRXOGILQGWKH PLQLPXP QXPEHUV RI KLJKRUGHU ELWV WKDW LV HQRXJK WR SHUIRUPWKHSUHGLFWLRQ:HFRQGXFWH[SHULPHQWVWRILQGWKH RSWLPDO FKRLFH IRU GLIIHUHQW OD\HUV E\ YDU\LQJ WKH ZLGWK RI KLJKRUGHUELWV1RWHWKDWQRWRQO\WKHLQSXWQHXURQVFRXOGEH VSOLWDVKLJKELWVDQGORZELWVWRSURYLGHWKHSUHGLFWLRQWKH KLJKRUGHUORZRUGHUELWVSOLWWLQJDOVRLV DSSOLFDEOHWR ILOWHU ZHLJKWV : 2XU FKDUDFWHUL]DWLRQ LQ )LJ FRQVLGHUV ERWK LQSXWDQGZHLJKWVSOLWWLQJPHWKRGV :HDSSO\LQSXWVSOLWWLQJDQGZHLJKWVSOLWWLQJPHWKRGVWR &219 DQG )&1 UHVSHFWLYHO\ DQG UHSRUW WKH SHUIRUPDQFH LPSURYHPHQW DQG WKH QXPEHU RI PHPRU\ DFFHVV ZLWK GLIIHUHQW VSOLWWLQJ RSWLRQV ZHLJKW VSOLWWLQJ LQ &219 DQG )&1 &219B:)&1B: LQSXW VSOLWWLQJ LQ &219 DQG )&1 &219B,)&1B, DQGDFRPELQDWLRQRIWZRVSOLWWLQJ PHWKRGV &219B,)&1B: &RPSDULQJ 2SWLRQ &219B:)&1B: DQG2SWLRQ &219B,)&1B, ZH REVHUYH WKDW LQSXW VSOLWWLQJ PHWKRG FDQ DFKLHYH KLJKHU VSHHGXS VLQFH LW UHTXLUHV OHVV KLJKRUGHU ELWV ZKLOH ZHLJKW VSOLWWLQJ PHWKRG KDV DQ DGYDQWDJH RI PHPRU\ DFFHVV HIILFLHQF\ 7KLV LV EHFDXVH ZHLJKW VSOLWWLQJ FDQ E\SDVV WKH ORDGLQJRIWKHZHLJKWVUHODWHGWRWKHL(21VLQ)&1OD\HUV
7$%/(, 48$17,=,1*,1387$1':(,*+7,17+(35(',&7,21 &219 ,QSXW6SOLWWLQJ
$OH[QHW 9** 9** 9** 1L1
)&1 :HLJKW 6SOLWWLQJ 1$
6TX )RUDFFXUDF\FRPSDUHGZLWKWKHEDVHOLQHSURILOLQJWKHQXPEHURI KLJKRUGHUELWVUHTXLUHGLQWKHSUHGLFWLRQ
&219B,)&1B:
6SHHGXS
&Ϭ
&ŝůƚĞƌϬ
$OH[1HW 9** 9**
$OH[1HW 9** 9**
/ͼtϭ
/ͼtW>
/ͼtϮ
&ϭ
&ŝůƚĞƌϭ
&Ϯ
&ŝůƚĞƌϮ
Ă
&WŽ
&ϬΎ/tϬ
/ŶƉƵƚ&ĞĂƚƵƌĞDĂƉ
)LJXUH6SOLWWLQJ:HLJKWYV,QSXWDPRQJ'LIIHUHQW/D\HUV
&ŝůƚĞƌƐ
&RQVLGHULQJ &219 LV FRPSXWHLQWHQVLYH DQG )&1 LV PHPRU\LQWHQVLYHZHFKRRVHLQSXWVSOLWWLQJLQ&219 DQG ZHLJKWVSOLWWLQJLQ)&1 2SWLRQ $VVKRZQLQ)LJWKLV FRPELQDWLRQ RI WZR VSOLWWLQJ PHWKRGV KDV UHODWLYHO\ KLJKHU VSHHGXSDQGIHZHUPHPRU\DFFHVV WKHEHVWFRPSUHKHQVLYH SHUIRUPDQFH 7$%/(,OLVWVWKHOHDVWQXPEHUVRIKLJKRUGHU ELWVLQGLIIHUHQWOD\HUVDPRQJYDULRXVQHWZRUNVIRUSUHGLFWLRQ ZLWKRXWDFFXUDF\ORVV $OWKRXJK WKH SUHGLFWRU FDQXVH D IHZ KLJKRUGHU ELWV WR SHUIRUPWKHSUHGLFWLRQWKHWZRVWDJHGHVLJQVKRZQLQ)LJ LVIDUIURPIXOO\OHYHUDJLQJWKLVRSSRUWXQLW\)LUVWWKHQXPEHU RIORZRUGHUELWVLQ([HFXWLRQ6WDJHLVJUHDWO\ODUJHUWKDQWKH QXPEHU RI KLJKRUGHU ELWV LQ 3UHGLFWLRQ 6WDJH :RUVH WKH LQSXWVSOLWWLQJYDULHVDFURVVGLIIHUHQWOD\HUV,IZHXVHSDUDOOHO PXOWLSOLHUV WR SHUIRUP WKH EDVLF 0$& RSHUDWLRQV LQ WKHVH WZR VWDJHV GLIIHUHQW QHWZRUNV GHPDQG WKHLU VSHFLILF PXOWLSOLHU GHVLJQV ZLWK GLIIHUHQW ELWZLGWKV WR PD[LPL]H HQHUJ\ HIILFLHQF\ ZKLFK LV LPSUDFWLFDO )XUWKHUPRUH WKH UDWLRRIWKHQXPEHURI0$&RSHUDWLRQV nMAC RQWKHWZR VWDJHV nMACExecutionnMACPrediction YDULHV VLJQLILFDQWO\ DPRQJGLIIHUHQWOD\HUVDVVKRZQLQ)LJ (YHQWKRXJKZH FDQXVHWZRVSHFLILFPXOWLSOLHUVIRUHDFKVWDJH LHELWVIRU 3UHGLFWLRQ6WDJHDQGELWVIRU([HFXWLRQ6WDJH WKHXQHYHQ FRPSXWDWLRQDO UDWLR VWLOO FRPSOLFDWHV WKH WZRVWDJH SLSHOLQH GHVLJQ DQG UHVXOWV LQ SLSHOLQH EXEEOHV 7KXV WKH PLVPDWFK EHWZHHQ KDUGZDUH XWLOL]DWLRQ DQG GLIIHUHQW ELWZLGWK UHTXLUHPHQWVPRWLYDWHVXVWRGHVLJQDXQLIRUPDUFKLWHFWXUHWR VXSSRUWERWK3UHGLFWLRQDQG([HFXWLRQ6WDJHV
/ŶƉƵƚϮ Ă /ŶƉƵƚW>
&ϬΎ/tϮ
&ϬΎ/tϭ
&ϬΎ/tW>
K&DϬ
Ă &ϭΎ/tϬ
&ϭΎ/tϮ
&ϭΎ/tϭ
&ϭΎ/tW>
K&Dϭ
Ă
Ă &ŝůƚĞƌWŽ
/ŶƉƵƚϭ
Ă
Ă
/ͼtϬ
/ŶƉƵƚϬ
&219B,)&1B,
E 9** D $OH[1HW )LJXUH 8QHYHQ&RPSXWDWLRQDO5DWLRRQ7ZR6WDJHV
1RUP0HP$FFHVV
&219B:)&1B:
&ϮΎ/tϬ
Ă
&ϮΎ/tϭ
Ă &WŽΎ/tϬ
&ϮΎ/tϮ
Ă &WŽΎ/tϭ
&ϮΎ/tW>
K&DϮ
Ă Ă &WŽΎ/tϮ
&WŽΎ/tW>
KƵƚƉƵƚ&ĞĂƚƵƌĞDĂƉƐ
1HWZRUN
K&DWŽ
h^W ƌƌĂLJ
)LJXUH6FDOHRXW'HVLJQLQ3UHGLFWLRQ6WDJH
7KLVXQLIRUPVHULDOPXOWLSOLHUEDVHGDUFKLWHFWXUHFDQEH XVHG LQ ERWK SUHGLFWRU DQG H[HFXWRU UHJDUGOHVV RI WKH ELW ZLGWKRIWKHLULQSXWV7KHGLIIHUHQFHLQWKHELWZLGWKZLOOEH UHIOHFWHGE\WKHQXPEHURIUXQQLQJF\FOHV7DNHDLQSXW VSOLWWLQJLQELWIL[SRLQWIRUPDWDVDQH[DPSOH,WWDNHV F\FOHVIRUSUHGLFWLRQDQGF\FOHVIRUH[HFXWLRQ7KXVZLWK WKHVHULDOPXOWLSOLHUWKHUHGXFHGELWZLGWKFDQOHDGWRORZHU ODWHQF\ ,Q )LJ ZH GHSOR\ WKH VHULDO PXOWLSOLHU LQWR RXU WZR VWDJH EDVHG SURFHVVLQJ HOHPHQW DUFKLWHFWXUH USPE &RPSDUHGZLWKWKHGHVLJQVKRZQLQ)LJWKHSUHGLFWRUDQG WKH H[HFXWRU VKDUH D XQLIRUP VHULDO PXOWLSOLHU EDVHG DUFKLWHFWXUH7KLVXQLIRUPDUFKLWHFWXUHRYHUFRPHVWKHDERYH WZR GLVDGYDQWDJHV GHPDQGLQJ PXOWLSOH PXOWLSOLHU GHVLJQV DQGSLSHOLQHEXEEOHV:HXVHD)LQLWH6WDWH0DFKLQH Stage 6LJQDO WR FRQWURO WKH FRPSXWDWLRQ FRQYHUVLRQ EHWZHHQ 3UHGLFWLRQ6WDJHDQG([HFXWLRQ6WDJH:KHQWKHFXUUHQWVWDWH LV3UHGLFWLRQ6WDJHIHBsLVHQDEOHGDQGWKHUSPELVFRQILJXUHG DVWKHSUHGLFWRU8VLQJWKHSignXQLWWKHSUHGLFWLRQUHVXOWVFDQ EHFRQYHUWHGWR SRVLWLYH DQG QHJDWLYH DQGWKHQZURWH LQWR ReLU_Table 8VLQJ Comparator XQLW WKH SUHGLFWLRQ UHVXOWV FDQ EH XVHG WR ILQG WKH PD[LPXP RXWSXW QHXURQV ZKLFKDUHUHFRUGHGLQMAX_Table:HZLOOGHVFULEHGHWDLOV RI WKHVH WDEOHV LQ 6HFWLRQ ,9% $W ([HFXWLRQ 6WDJH USPE RQO\ SURFHVVHV WKH (21V EDVHG RQ ReLU_Table RU MAX_TableILBsUHODWHGWRHIIHFWXDORXWSXWQHXURQLVHQDEOHG DQG WKH UHPDLQLQJ FDOFXODWLRQ LV WKHQ SHUIRUPHG RQ WKH XQLIRUP DUFKLWHFWXUH )LQDOO\ WKH ILQDO YDOXHV RI (21V DUH REWDLQHGE\DGGLQJWKHUHVXOWVRI3UHGLFWLRQ6WDJH
B. A Uniform, Serial Multiplier (Architecture Reuse) $VGLVFXVVHGLQ6HFWLRQ,,,$WKHPXOWLSOLHULVWKHEDVLF XQLWLQERWKSUHGLFWRUDQGH[HFXWRU7KHYDULRXVQXPEHURI ELWVLQWKHVH PXOWLSOLHUV¶RSHUDQGV PRWLYDWHVXV WR GHVLJQD VHULDO PXOWLSOLHU EDVHG DUFKLWHFWXUH WR UHSODFH WKH SDUDOOHO PXOWLSOLHUDVVKRZQLQ)LJ
,9 29(5$//6&$/(287'(6,*1 ,Q 6HFWLRQ ,,, ZH SURSRVH USPE DV D EDVLF SURFHVVLQJ DUFKLWHFWXUH WKDW LV DEOH WR E\SDVV WKH FRPSXWDWLRQ RI
756
/ŶƉƵƚ&ĞĂƚƵƌĞDĂƉ
Ͳϭ ͲϬ͘ϵ ϭ
Ϭ
&Ϭ ϭ͘ϭ ϭ Ͳϭ Ͳϭ
&ϭ
/ͼt;Ϭ͕ϬͿ
/ͼt;Ϭ͕ϭͿ
/ͼt;Ϭ͕ϮͿ
/ͼt;Ϭ͕ϯͿ
ϭ Ϭ͘ϵ
Ϭ͘ϴ Ϭ͘ϱ
Ϭ͘ϯ Ϭ͘Ϯ
Ϭ͘ϵ Ϭ͘ϵ
ϭ Ϭ͘ϴ
Ϭ͘ϱ Ϭ͘ϱ
Ϭ͘ϳ Ϭ͘ϲ
Ϭ͘ϳ Ϭ͘ϲ
Ϭ͘Ϯ Ϭ͘ϯ
Ϭ͘Ϯ Ϭ͘ϱ
Ϭ͘ϳ Ϭ͘ϲ
Ϭ͘ϳ Ϭ͘ϲ
ϭ Ϭ͘ϱ
Ϭ͘ϴ Ϭ͘ϲ
Ϭ͘ϰ Ϭ͘ϴ
Ϭ͘Ϯ Ϭ͘ϯ
/ͼt;ϭ͕ϬͿ
/ͼt;ϭ͕ϭͿ
/ͼt;ϭ͕ϮͿ
/ͼt;ϭ͕ϯͿ
Ϯ
ϯ
Ϭ
ͲϬ͘ϴ ͲϬ͘ϴ Ϭ͘Ϯ
Ϭ
Ͳϭ
ϭ
Ϭ͘ϱ
Ͳϭ
LJ
ϭ
Ϭ͘Ϯ ͲϬ͘ϴ
K&DϬ Ϭ
ϭ
Ϯ
ϯ
Ϭ
Ϭ͘Ϯ
Ϭ͘ϰ ͲϬ͘ϴ Ϭ͘ϲ
ϭ
Ͳϭ
ͲϬ͘ϳ Ϭ͘Ϯ Ϭ͘ϵ
LJ
dž Kͬ/t ;Ϭ͕ϬͿ
;Ϭ͕ϭͿ
;Ϭ͕ϮͿ
;Ϭ͕ϯͿ
;ϭ͕ϬͿ
;ϭ͕ϭͿ
;ϭ͕ϮͿ
;ϭ͕ϯͿ
&Ϭ
Ϭ
Ϭ
ϭ
Ϭ
ϭ
Ϭ
Ϭ
Ϭ
&ϭ
Ϭ
ϭ
Ϭ
Ϭ
Ϭ
Ϭ
Ϭ
ϭ
DyͺdĂďůĞ
dž
Kͬ/t ;Ϭ͕ϬͿ
K&Dϭ
;Ϭ͕ϭͿ
;Ϭ͕ϮͿ
;Ϭ͕ϯͿ
;ϭ͕ϬͿ
;ϭ͕ϭͿ
;ϭ͕ϮͿ
;ϭ͕ϯͿ
&Ϭ
Ϭ
Ϭ
ϭ
Ϭ
ϭ
ϭ
Ϭ
Ϭ
&ϭ
ϭ
ϭ
Ϭ
ϭ
Ϭ
Ϭ
ϭ
ϭ
ZĞ>hͺdĂďůĞ
;ĂͿŶ džĂŵƉůĞŽĨŽŶǀŽůƵƚŝŽŶĂůKƉĞƌĂƚŝŽŶƐ /ͼt;Ϭ͕ϬͿ &Ϭ Ͳϭ
ͲϬ͘ϵ
ϭ
Ϭ
ϭ͘ϭ
ϭ
Ͳϭ
Ͳϭ
/ͼt;Ϭ͕ϭͿ
ϭ
Ϭ͘ϵ
Ϭ͘ϴ
Ϭ͘ϱ
ϭ
Ϭ͘ϴ
Ϭ͘ϱ
Ϭ͘ϱ
/
/
/
&ϭ
/ͼt;Ϭ͕ϮͿ &Ϭ Ͳϭ
ͲϬ͘ϵ
ϭ
Ϭ
ϭ͘ϭ
ϭ
Ͳϭ
Ͳϭ
Ϭ͘ϰ
Ϭ͘ϯ Ϭ͘Ϯ
/ͼt;Ϭ͕ϯͿ Ϭ͘ϵ Ϭ͘ϵ
Ϭ͘ϳ Ϭ͘ϲ
Ϭ͘ϳ Ϭ͘ϲ
&ϭ
LJĐůĞϬͲϳ
&Ϭ
Ϭ͘Ϯ
/
/
/
Ͳϭ
ͲϬ͘ϵ
ϭ
Ϭ
ϭ͘ϭ
ϭ
Ͳϭ
/ͼt;ϭ͕ϬͿ
/ͼt;ϭ͕ϭͿ
Ϭ͘Ϯ Ϭ͘ϯ
Ϭ͘Ϯ Ϭ͘ϱ
ϭ
Ϭ͘ϴ Ϭ͘ϲ
Ϭ͘ϱ
Ϭ͘ϱ
/
/
/
Ͳϭ
&ϭ
/ͼt;ϭ͕ϮͿ
/ͼt;ϭ͕ϯͿ
Ϭ͘ϳ Ϭ͘ϲ
Ϭ͘ϳ Ϭ͘ϲ
Ϭ͘ϰ Ϭ͘ϴ
Ϭ͘Ϯ Ϭ͘ϯ
&Ϭ Ͳϭ
ͲϬ͘ϵ
ϭ
Ϭ
ϭ͘ϭ
ϭ
Ͳϭ
Ͳϭ
&ϭ
LJĐůĞϭϲͲϮϯ
LJĐůĞϴͲϭϱ
&Ϭ
/ͼt;Ϭ͕ϮͿ
Ϭ͘Ϯ
&Ϭ
Ϭ͘ϯ
Ͳϭ
&Ϭ
Ϭ͘ϳ
/
/
LJĐůĞϬ /ͼt;ϭ͕ϬͿ /ͼt;Ϭ͕ϮͿ ϭ
ϭ
ͲϬ͘ϯ
LJĐůĞϭ /ͼt;Ϭ͕ϭͿ /ͼt;ϭ͕ϯͿ Ϭ͘ϱ
Ͳϭ
LJĐůĞϰ
ϭ͘ϯϳ
&ϭ
Ϭ͘ϴϴ
Ϭ͘ϳϳ
/ͼt;ϭ͕ϬͿ
LJĐůĞϮ /ͼt;Ϭ͕ϮͿ
Ϭ͘ϱ
Ϭ͘ϲ
&ϭ
/ͼt;Ϭ͕ϭͿ
Ϭ͘ϵ
>ĞŐĞŶĚŽĨh^W ƐƚĂƚƵƐ
/
/ĚůĞ
Ϭ͘ϮϮ
&ŝŶĂůZĞƐƵůƚ
Ϭ͘ϴϳ
/ŶƚĞƌŵĞĚŝĂƚĞ ZĞƐƵůƚ
Ϭ͘ϲ
ͲϬ͘ϰϳ
ͲϬ͘ϰϴ
&ϭ
&Ϭ
ŽŵƉƵƚŝŶŐ
/ͼt;ϭ͕ϯͿ
Ϭ͘ϱ
/ͼt;Ϭ͕ϭͿ
LJĐůĞϯ /ͼt;ϭ͕ϯͿ
Ϭ͘ϱ
Ͳϭ
/
WKсϮ W>сϮ
&Ϭ
Ϭ͘ϯ
/ͼt
/ͼt
Ϭ
Ϭ͘ϮϮ
/
ϭ
Ϭ
Ϭ͘ϱϯ Ͳϭ
ϭ͘ϯϴ
&Ϭ
Ϭ͘Ϯ
/
ͲϬ͘ϵ
&ϭ
&Ϭ
Ϭ͘Ϯ
ϭ
/ͼt;Ϭ͕ϮͿ
Ϭ͘ϯ
ϭ
&ϭ
&Ϭ
Ϭ͘ϳ
/ͼt;ϭ͕ϬͿ
ͲϬ͘ϵ
ͲϬ͘Ϯ ϭ͘ϭ
&ϭ
&ϭ
/ͼt;ϭ͕ϯͿ
Ϭ͘ϴ
Ͳϭ
ϭ͘ϭ
&Ϭ
/ͼt;Ϭ͕ϭͿ
W/сϭ E>ƐсϮ
LJĐůĞϮϰͲϯϭ
;ďͿ/ŵƉůĞŵĞŶƚĂƚŝŽŶŽŶƚŚĞdǁŽͲĚŝŵĞŶƐŝŽŶ^ŚĂƌŝŶŐƌĐŚŝƚĞĐƚƵƌĞ /ͼt;ϭ͕ϬͿ
ƌƌĂLJŽŶĨŝŐƵƌĂƚŝŽŶƐ
Ϭ͘ϱ
Ϭ͘Ϯ
Ϭ͘ϰ
Ϭ͘ϵ
Ͳϭ
Ϭ͘ϴϴ
LJĐůĞϱ
ϭ͘ϭϳ
&ϭ
LJĐůĞϲ
;ĐͿ/ŵƉůĞŵĞŶƚĂƚŝŽŶŽŶƚŚĞKŶĞͲĚŝŵĞŶƐŝŽŶ^ŚĂƌŝŶŐƌĐŚŝƚĞĐƚƵƌĞ
LJĐůĞϳ
&ϭ
LJĐůĞϴ
)LJXUH0DSSLQJ0D[3WRWKH6FDOHRXW'HVLJQ
LQHIIHFWXDO RXWSXW QHXURQV L(21V DW VLQJOH QHXURQ OHYHO +RZHYHU VLQFH '11 FRPSXWDWLRQ XVXDOO\ LQYROYHV D ODUJH QXPEHU RI QHXURQV WKH VFDODELOLW\ RI USPE LV HYHQ PRUH FUXFLDO :HSUHVHQWDVLPSOHVFDOHRXWGHVLJQDVVKRZQLQ)LJ ZKHUH WKH USPE XQLW LV GXSOLFDWHG LQ WZR GLPHQVLRQV WR LPSURYH WKH SURFHVVLQJ WKURXJKSXW 6LQFHUSPE XVHV D ELW VHULDOPXOWLSOLHUWRSHUIRUPWKHEDVLFPXOWLSOLFDWLRQLWWDNHV PXOWLSOHF\FOHVWRDFFRPSOLVKRQHPXOWLSOLFDWLRQ,QWKHZRUVW FDVH USPE FDQ WDNH PD[LPXP NLBs (maxNLBs) F\FOHV WR FDOFXODWH D SURGXFW LQ ([HFXWLRQ 6WDJH 7R PDLQWDLQ WKH FRPSDUDEOHSHUIRUPDQFHDVREWDLQHGE\DSDUDOOHOPXOWLSOLHU ZHQHHGWRVLPXOWDQHRXVO\SURFHVVPLRXWSXWQHXURQVLQRQH RXWSXWIHDWXUHPDS 2)0 ZKHUHPLLVODUJHUWKDQmaxNLBs 7KHUHIRUH WKH SDUDOOHOLVP JUDQXODULW\ RI WKH KRUL]RQWDO GLUHFWLRQLQ)LJLVPL7RPDWFKWKHSURFHVVLQJWKURXJKSXW RI RWKHU VWDWHRIWKHDUW DFFHOHUDWRUV VXFK DV 'LDQ1DR ZH DOVR XQUROO YDULRXV ILOWHUV VR WKDW RXU DFFHOHUDWRU FDQ VLPXOWDQHRXVO\SHUIRUPPXOWLSOH2)0V$VVKRZQLQ)LJ WKH YHUWLFDO SDUDOOHOLVP JUDQXODULW\ LV WKH QXPEHU RI 2)0V PO ,QWKLVVHFWLRQZHH[SORUHWKHFKDOOHQJHVRILQWHJUDWLQJ RXU WZRVWDJH EDVHG FRPSXWLQJ SDWWHUQ WR WKH VFDOHRXW DUFKLWHFWXUHDQGSURSRVHRXUVROXWLRQV
A. The Impact of Scale-out on Prediction Stage :HILUVWH[DPLQHZKHWKHU WKHSUHGLFWLRQVWDJHFRXOG EH VHDPOHVVO\ LPSOHPHQWHG RQ WKH VFDOHRXW GHVLJQ ZLWKRXW LQWURGXFLQJ RYHUKHDGV $V VKRZQ LQ )LJ HDFK USPE LV UHVSRQVLEOH IRU WKH FDOFXODWLRQ RI RQH RXWSXW QHXURQ 7KXV WKLVUSPEDUUD\FDQSUHGLFWPLൈPORXWSXWQHXURQVLQSDUDOOHO 7KHUH H[LVW WZR RSSRUWXQLWLHV IRU GDWD VKDULQJ WKH IHWFKHG GDWDFRXOG EH VKDUHGDPRQJUSPEVDIWHU RQO\ RQH PHPRU\ ORDG LQ WKLV USPE DUUD\ ZKLFK FRXOG EH OHYHUDJHG E\ 3UHGLFWLRQ 6WDJH WR PLWLJDWH PHPRU\ DFFHVV RYHUKHDG 7KH ILUVW LV ILOWHU VKDULQJ 6LQFH DOO RXWSXW QHXURQV LQ RQH URZ EHORQJWRWKHVDPH2)0DQGHDFK2)0FRUUHVSRQGVWRRQH GHGLFDWHG ILOWHU WKH USPEV LQ RQH URZ FDQ VKDUH WKH VDPH ILOWHUGDWD7KHVHFRQGLVLQSXWVKDULQJ$OWKRXJKWKHRXWSXW QHXURQV WKDW DUH SURFHVVHG LQ WKH VDPH FROXPQ EHORQJ WR GLIIHUHQW2)0V WKH\DUHORFDWHGDW WKHVDPHSODFHRI WKHLU 2)0V1RWHWKDWWKHRXWSXWQHXURQVZLWKWKHVDPHORFDWLRQ DUH FDOFXODWHG IURP WKH VDPH LQSXW QHXURQV 7KXV DOO WKH USPE LQ RQH FROXPQ VKDUH WKH VDPH LQSXW GDWD HJ $V VKRZQLQ)LJWKRXJKDOOJUHHQQHXURQVEHORQJWR2)0 2)0«2)03RUHVSHFWLYHO\WKH\DOOORFDWHDWWKHWRSOHIW FRUQHU RI HDFK RXWSXW IHDWXUH PDS DQG DUHFDOFXODWHG XVLQJ
757
WKHVDPHLQSXWZLQGRZ ,: 6LQFHDOOWKHRXWSXWQHXURQV QHHGWREHSUHGLFWHGLQ3UHGLFWLRQ 6WDJHWKHVH WZRVKDULQJ RSSRUWXQLWLHV FDQ EH IXOO\H[SORLWHG :LWKWKH WZRNLQGV RI GDWD VKDULQJ LQSXW DQG ILOWHU PL LQSXWV DQG PO ILOWHUV FDQ PDNHWKHUSPEDUUD\EHIXOO\XWLOL]HG $IWHUWKHSUHGLFWLRQLQ3UHGLFWLRQ6WDJHZHFDQREWDLQWKH ORFDWLRQLQIRUPDWLRQRI(21V$QGWKHSUHGLFWLRQUHVXOWVDUH ZULWWHQLQWRWKHMax_TableDQGReLU-TableUHVSHFWLYHO\LQ )LJ D (DFKHOHPHQWLQWKHVHWDEOHVFRUUHVSRQGVWRRQH RXWSXW QHXURQ 9DOXH PHDQV LWV FRUUHVSRQGLQJ RXWSXW QHXURQLVLQHIIHFWXDOZKLOHLQGLFDWHVHIIHFWXDO
7$%/(,,7+(180%(52)(216 /D\HU $OH[/D\HU 9**/D\HU
3HU&RRUGLQDWH 0HDQ 9DU
3HU2)0 0HDQ 9DU
DQGILOWHUVKDULQJZHQHHGPLൈPOILOWHUVDQGLQSXWVWRNHHS DOOWKHUSPEVEXV\6LQFHWKH(21VDUHUDQGRPO\ORFDWHGLQ WKH 2)0V DOO WKHLU UHTXLVLWH LQSXWV DQG ILOWHUV DUH DOVR UDQGRPO\ORFDWHGLQWKHLQSXWDQGILOWHUPHPRU\VRWKDWWKH\ FDQQRW EH IHWFKHG WRJHWKHU ,Q WKH ZRUVW FDVH LW UHTXLUHV PL ൈPO F\FOHV WR ORDG DOO WKH GDWD ZKLOH USPE DUUD\ RQO\ QHHGVNLBsF\FOHVWRSURFHVVWKHVHGDWD6LQFHWKHGDWDORDGLQJ WLPH PLൈPO LVODUJHUWKDQWKHGDWDSURFHVVLQJWLPH NLBs PHPRU\DFFHVVEHFRPHVWKHERWWOHQHFNRIWKHDFFHOHUDWRU 7RDOOHYLDWHWKHSUHVVXUHRIPHPRU\DFFHVVZHVWLOOQHHG WRDGRSWRQHNLQGRIGDWDVKDULQJLQ([HFXWLRQ6WDJH7KHUH DUHWZRFKRLFHV LQSXWRUILOWHUVKDULQJ IRU0D[3DQG5H/8 Max-pooling $OWKRXJKWKHGLVWULEXWLRQRI(21VLVUDQGRP LQGLIIHUHQW2)0VWKHQXPEHURIWKH(21VLQHDFK2)0VLV QHDUO\WKHVDPHDQGFDQEHFDOFXODWHGEDVHGRQWKHQHWZRUN DUFKLWHFWXUHSDUDPHWHUVVXFKDVPD[SRROLQJVL]HDQGVWULGH VL]H )RU LQVWDQFH DV VKRZQ LQ )LJ D WKH SHUFHQW RI (21VLQRQH2)0LVDERXWIRUWKHൈ0D[3OD\HUZLWK VWULGH VL]H 7KH VWDEOH QXPEHU RI (21V DPRQJ 2)0V LQGLFDWHVWKDWWKHFDOFXODWLRQWLPHRIHDFK2)0LVVLPLODUWR RWKHUV0RUHRYHUDOOWKH(21VLQWKHVDPH2)0FRUUHVSRQG WR WKH VDPH ILOWHU 7KHUHIRUH ZH FKRRVH ILOWHU VKDULQJ IRU 0D[3DQGWKHLGOHQHVVFDQEHDYRLGHGVLQFHWKHFRPSXWDWLRQ WLPHRIGLIIHUHQW2)0VLVEDODQFHG ReLU˖'LIIHUHQW IURP WKH 0D[3 WKH QXPEHU RI (21V LQ HDFK2)0YDULHVIRU5H/8,IZHVWLOOFKRRVHILOWHUVKDULQJ IRU 5H/8 WKH USPEV LQ RQH FROXPQ PD\ KDYH GLIIHUHQW ZRUNORDGV ZKLFK UHVXOWV LQ WKH LGOHQHVV RI USPE ZKHQ LWV ZRUNORDG LVOLJKW)RUWXQDWHO\WKHVXPPDWLRQRI WKH(21V ZLWKWKHVDPHFRRUGLQDWHDFURVVGLIIHUHQW2)0VLVQHDUO\WKH VDPH)RUH[DPSOHDVVKRZQLQ7$%/(,,WKHYDULDQFHRQO\ DFFRXQWVIRURILWVPHDQIRUWKHQXPEHURI(21VZLWK WKHVDPHFRRUGLQDWH6LQFHRXWSXWQHXURQVEHORQJLQJWRWKH VDPHORFDWLRQLVFDOFXODWHGIURPWKHVDPHLQSXWZHFKRRVH LQSXWVKDULQJIRU5H/8 3) Max-pooling: Filter Sharing 6LQFHZHXVHILOWHUVKDULQJIRU0D[3WKHPHPRU\DFFHVV UHODWHG WR LQSXW EHFRPHV WKH PDLQ ERWWOHQHFN LQ ([HFXWLRQ 6WDJH 7R WDFNOH WKLV FKDOOHQJH ZH GHVLJQ PL LQSXW EXIIHU FRQWUROOHUV DV VKRZQ LQ )LJ (DFK EXIIHU FRQWUROOHU LV UHVSRQVLEOHIRUDVVLJQLQJWKHLQSXWGDWDWRWKHUSPEVLQRQH FROXPQ $IWHU WKH LQSXW GDWD LV ORDGHG IURP PHPRU\ LW ZLOO EH FDFKHGLQD3LQJSRQJ*OREDO%XIIHU 6WHSķ GDWDORDGLQJ 7KLV JOREDO EXIIHU LV EXLOW XSRQ D EDVLF LGHD RI GRXEOH EXIIHULQJ LQ ZKLFK GRXEOH EXIIHUV DUH RSHUDWHG LQ D SLQJ SRQJPDQQHUWRRYHUODSGDWDORDGLQJZLWKFRPSXWDWLRQ7KHQ DFFRUGLQJ WR WKH MAX_Table UHFRUGV VWHS ĸ EXIIHU FRQWUROOHUV GLVWULEXWH WKH LQSXW GDWD WR WKHLU FRUUHVSRQGLQJ USPEVLQWKHIROORZLQJF\FOHV VWHSĹ GDWDDVVLJQPHQW
B. Challenges and Optimizations of Execution Stage :H WKHQ H[DPLQH WKH FKDOOHQJHV RI SDUDOOHOL]LQJ ([HFXWLRQ6WDJHLQWKHVFDOHRXWGHVLJQ7KURXJKDQDO\]LQJ WKH UDQGRPQHVV ZH PDQDJH WR ILQG WKH XQGHUO\LQJ UHJXODULWLHV RI WKH QXPEHU DQG WKH GLVWULEXWLRQ RI (21V LQ 0D[3 DQG 5H/8 UHVSHFWLYHO\ :H WKHQ SURSRVH FRUUHVSRQGLQJGDWDVKDULQJSROLFLHVIRU0D[3DQG5H/8DQG YDOLGDWHRXUGHVLJQ 1) Challenges Caused by Randomness and Sparsity of EONs Randomness of EONs 6LQFHWKHL(21VFRXOGEHE\SDVVHG EDVHGRQSUHGLFWLRQUHVXOWVLQ3UHGLFWLRQ6WDJHZHRQO\QHHG WR SHUIRUP WKH UHPDLQLQJ FDOFXODWLRQ UHODWHG WR WKH (21V UHFRUGHGLQWKHMax_TableDQGReLU_Table+RZHYHUWKH GLVWULEXWLRQRI(21VLQ2)0VLVLUUHJXODU,Q0D[3OD\HUV WKH GLVWULEXWLRQ RI (21V LV UDQGRP LQ HDFK 2)0 )RU LQVWDQFH DV VKRZQ LQ )LJ D WKH (21V LH PD[LPXP RXWSXWQHXURQV DUHORFDWHGDW DQG LQ2)0ZKLOH WKH\ DUH ORFDWHG DW DQG LQ 2)0 7KLV UDQGRP GLVWULEXWLRQ DOVR DSSOLHV WR 5H/8 %HVLGHV GLVWULEXWLRQ WKH QXPEHURI(21VLQHDFK2)0LVGLIIHUHQWDVZHOOLQ5H/8 )RUH[DPSOHDVVKRZQLQ)LJ D WKHQXPEHURI(21VLQ 2)0LVZKLOH2)0KDV(21V Idleness of USPE Array 7KHQDwYHVFDOHRXWGHVLJQVKRZQ LQ )LJ GRHV QRW ZHOOVXSSRUW L(21 VNLSSLQJ 6LQFH WKH ILOWHU DQG LQSXW GDWD DUH VKDUHG DPRQJ WKH VDPH URZ RU FROXPQRIUSPEVWKHUSPEKDVWREHVHWDVLGOHZKHQLWLV DVVLJQHG DQ L(21 ZKLOH LWV QHLJKERU LV DVVLJQHG DQ (21 :LWKWKHQDwYHGHVLJQWKHQXPEHURILGOHUSPEVHTXDOVWRWKH QXPEHURIL(21V)RUH[DPSOHRIUSPEVDUHLGOHIRU 0D[3DVVKRZQLQ)LJ E &RQVHTXHQWO\WKHDSSURDFKRI SUHGLFWLRQ ZLOO PHUHO\ FRQWULEXWH WR WKH SRZHU VDYLQJ RI DFFHOHUDWRUE\SRZHUJDWLQJLGOHUSPEVZKLOHWKHVLJQLILFDQW EHQHILWWRUHGXFHWKHH[HFXWLRQWLPHZLOOQRWEHREWDLQHGLQ WKLV QDwYH DUFKLWHFWXUH 7R LPSURYH WKH SHUIRUPDQFH RI ([HFXWLRQ6WDJHZHVKRXOGHIIHFWLYHO\E\SDVVWKHSURFHVVRI L(21V DFFRUGLQJ WR MAX_Table DQG ReLU_Table DQG HQDEOHWKHUSPEDUUD\WRSHUIRUP(21VZLWKRXWLGOHQHVV 2) Data Sharing Policy in Execution stage 2QHPHWKRGWRLQFUHDVHWKHXWLOL]DWLRQRIUSPELVQRWWR OHYHUDJH WKH VKDULQJ RSSRUWXQLWLHV DV PHQWLRQHG LQ 6HFWLRQ ,9$ :LWKRXWVKDULQJWKHLQSXWGDWD DQG ILOWHUZLWKLQ SHHU USPEV LQ WKH VDPH URZ RU FROXPQ HDFK USPE FDQ ORDG LWV RZQLQSXWDQGILOWHURQGHPDQGEDVHGRQLWVDVVLJQHG(21 $OWKRXJKWKLVPHWKRGFDQDYRLGWKHLGOHQHVVRIUSPELWDOVR FKDOOHQJHV WKH PHPRU\ DFFHVV FDSDFLW\ :LWKRXW WKH LQSXW
758
WŝŶŐͲƉŽŶŐ'ůŽďĂůƵĨĨĞƌ
/ŶƉƵƚϬ
^ƵďͲƌĞŐŝŽŶ
ʒ
Kͬ/t
&Ϭ &ϭ
DĂdžƉŽŽůŝŶŐƐŝnjĞhW>
ʔ Ϭ
ϭ
Ϯ
;Ϭ͕ϬͿ ;Ϭ͕ϭͿ ;Ϭ͕ϮͿ ;Ϭ͕ϯͿ ;ϭ͕ϬͿ ;ϭ͕ϭͿ ;ϭ͕ϮͿ ;ϭ͕ϯͿ Ϭ Ϭ ϭ Ϭ ϭ Ϭ Ϭ Ϭ Ϭ ϭ Ϭ Ϭ Ϭ Ϭ Ϭ ϭ
ʓ Ă
&Ϭ
&ŝůƚĞƌϬ
&ϭ
&ŝůƚĞƌϭ
Ă
&Ϯ
&ŝůƚĞƌϮ
Ă
W>
/ŶƉƵƚƵĨĨĞƌŽŶƚƌŽůůĞƌƐ
Ă
K&DϬ K&Dϭ Ă
&ŝůƚĞƌƐ
K&DϮ
Ă &ŝůƚĞƌWŽ
/ŶƉƵƚW>
h^W ƌƌĂLJ
DyͺdĂďůĞ
Ă
W>/ŶƉƵƚƐ
&WŽ
/ŶƉƵƚϮ Ă
ʕ
Ă
Ă
Ă
Ă Ă
Ă
KƵƚƉƵƚ&ĞĂƚƵƌĞDĂƉƐ
Ă
/ŶƉƵƚϭ
Ă
&ŝůƚĞƌ ƵĨĨĞƌŽŶƚƌŽůůĞƌƐ
Ă
Ϭ
ϭ
Ă
Ϯ
Ă
ʒ Ă
WK&ŝůƚĞƌƐ
h^W ƌƌĂLJ
)LJXUH6FDOHRXW'HVLJQIRU0D[3
ʓ ZĞ>hͺdĂďůĞ
Ă Ă
EĨ WŝŶŐͲƉŽŶŐ'ůŽďĂůƵĨĨĞƌ
W>
Ă
ʔ
K&DWŽ
ʕ
Ă
Kͬ/t
;Ϭ͕ϬͿ ;Ϭ͕ϭͿ ;Ϭ͕ϮͿ ;Ϭ͕ϯͿ ;ϭ͕ϬͿ ;ϭ͕ϭͿ ;ϭ͕ϮͿ ;ϭ͕ϯͿ
&Ϭ
Ϭ
Ϭ
ϭ
Ϭ
ϭ
ϭ
Ϭ
Ϭ
&ϭ
ϭ
ϭ
Ϭ
ϭ
Ϭ
Ϭ
ϭ
ϭ
)LJXUH6FDOHRXW'HVLJQIRU5H/8
FRXOG QRW VNLS RYHU WKH L(21V ZKLFK UHVXOWV LQ LGOH USPE)LJ F RQO\NHHSVWKHILOWHUVKDULQJDPRQJUSPELQ RQHURZDQGLQWURGXFHVEXIIHUFRQWUROOHUVWRDVVLJQWKHLQSXW GDWD:LWKWKHEXIIHUFRQWUROOHUDOOWKHUSPEVFDQREWDLQLWV LQSXW ZLWKLQ WZR F\FOHV &RQVLGHULQJ WKH WZRF\FOH SURFHVVLQJWLPH NLBs DOOWKHUSPEVFDQREWDLQLWVQHZLQSXW GDWDZKHQILQLVKLQJWKHSURFHVVLQJRIFXUUHQWLQSXWGDWD7KXV WKLVUSPEDUUD\FDQEHIXOO\XWLOL]HGDIWHUF\FOH6LQFHWKH FDOFXODWLRQRIRXWSXWQHXURQLQFOXGHVIRXU0$&RSHUDWLRQV DQG RQH PXOWLSOLFDWLRQ QHHGV WZR F\FOHV XVLQJ WKH VHULDO PXOWLSOLHUWKLVUSPEDUUD\WDNHV [ F\FOHVWRILQLVKDOO WKHSURFHVVLQJ 4) ReLU: Input Sharing 6LQFH ZH XVH LQSXW VKDULQJ IRU 5H/8 WKH ILOWHUUHODWHG PHPRU\DFFHVVEHFRPHVWKHPDLQERWWOHQHFN+HUHZHUHXVH WKH SLQJSRQJ JOREDO EXIIHU DQG WKH EXIIHU FRQWUROOHUV DV GHVFULEHG LQ 6HFWLRQ ,9% WR RYHUFRPH WKLV ERWWOHQHFN &RPSDUHG ZLWK WKH GHVLJQ IRU 0D[3 DOO WKH USPEV LQ RQH FROXPQ VKDUH WKH VDPH LQSXW DQG HDFK EXIIHU FRQWUROOHU LV UHVSRQVLEOHIRUILOWHUGLVWULEXWLRQDPRQJUSPEVEHORQJLQJWR RQHFROXPQVKRZQLQ)LJ 6LPLODUO\WRIXOO\XWLOL]HWKHUSPEDUUD\RXUGHVLJQPXVW PHHW WKH UHTXLUHPHQWV RI GDWD DVVLJQPHQW DQG ORDGLQJ GLVFXVVHGLQ6HFWLRQ,9%1H[WZHGLVFXVVWKHWLPHRIGDWD DVVLJQPHQWDQGORDGLQJIRU5H/8 Data Assignment:6LQFHHDFKEXIIHUFRQWUROOHULVUHVSRQVLEOH IRUPOUSPEsDQGHDFKUSPEUHTXLUHVDGLIIHUHQWILOWHUXQGHU WKHZRUVWFDVHLWWDNHVPOF\FOHVWRILQLVKWKHGDWDDVVLJQPHQW 7RHQVXUHPHPRU\FRQWUROOHUFDQSURYLGHHQRXJKILOWHUVIRU HDFKUSPEWKHF\FOHVRIGDWDDVVLJQPHQWVKRXOGEHOHVVWKDQ WKDWRIGDWDSURFHVVLQJDV
7RNHHSDOOWKHUSPEVEXV\WKHDERYHGHVLJQVKRXOGPHHW WKH IROORZLQJ UHTXLUHPHQWV VR WKDW LW FDQ RYHUFRPH WKH ERWWOHQHFN RI PHPRU\ DFFHVV )LUVW WKH WLPH RI GDWD DVVLJQPHQW VKRXOG EH VKRUWHU WKDQ WKDW RI GDWD SURFHVVLQJ 6HFRQGWKHWLPHRIORDGLQJGDWDWRJOREDOEXIIHUDOVRVKRXOG EHVKRUWHUWKDQWKDWRIGDWDSURFHVVLQJ(DFKUSPEWDNHVWKH ELWZLGWKRILQSXW NLBs F\FOHVWRILQLVKWKHGDWDSURFHVVLQJ XVLQJ WKH VHULDO PXOWLSOLHU :H H[SORUH WKH GHWDLOHG WLPH PRGHORIGDWDDVVLJQPHQWDQGORDGLQJDQGYDOLGDWHWKDWRXU GHVLJQVDWLVILHVWKHDERYHUHTXLUHPHQWV Data Assignment:)RU0D[3OD\HUVWKHUHPXVWH[LVWDWOHDVW RQH(21LQHDFKVXEUHJLRQZLWKWKHVL]HRImaxpooling_size HJൈLQ)LJ LQWKH2)07KHUHIRUHZHDVVLJQRQH VXEUHJLRQWRDUSPE7KHUSPEFDOFXODWHVWKH(21LQWKLV UHJLRQ,QWKHZRUVWFDVHDOOORFDWLRQVZLWKLQWKHVXEUHJLRQ FDQ KDYH WKH (21 ,Q WKLVVLWXDWLRQ DOOUSPEV LQ WKH VDPH FROXPQQHHGmaxpooling_sizeLQSXWV6LQFHEXIIHUFRQWUROOHU FDQ RQO\ EURDGFDVW RQH LQSXW HDFK WLPH LW ZLOO WDNH maxpooling_size F\FOHV WR ILQLVK WKH LQSXW DVVLJQPHQW 1RUPDOO\WKHmaxpooling_sizeLVVPDOOVXFKDVൈRUൈ %DVHG RQ 7$%/( , WKH NLBs LV DOZD\V ELJJHU WKDQ 7KHUHIRUHWKHWLPHRILQSXWDVVLJQPHQW maxpooling_size LV VKRUWHU WKDQ WKDW RI GDWD SURFHVVLQJ NLBs ZKLFK LQGLFDWHV WKH PHPRU\ FRQWUROOHU GHVLJQ LV FDSDEOH RI NHHSLQJ DOO WKH USPEVEXV\ Data Loading: 6LQFH DOO USPEV LQ RQH FROXPQ QHHGV maxpooling_sizeLQSXWVXQGHUWKHZRUVWFDVHWKHUSPEDUUD\ ZLWK PL FROXPQV GHPDQGV DW PRVW maxpooling_size ൈ PL LQSXWVZKRVHVL]HLVDOVRWKHVL]HRIFDFKHGLQSXWLQWKHJOREDO EXIIHU$VWKH JOREDOEXIIHUFDQORDG PLFRQVHFXWLYHLQSXWV HDFKWLPHLWQHHGVmaxpooling_sizeF\FOHVWRDFFRPSOLVKWKH GDWDORDGLQJ6LQFHNLBsLVELJJHUWKDQmaxpooling_sizeZLWK WKH SLQJSRQJ EXIIHU GHVLJQ WKH GDWD ORDGLQJ WLPH FDQ EH RYHUODSSHGZLWKWKHWLPHRIGDWDSURFHVVLQJ :HWDNHDVLPSOHH[DPSOH WZRILOWHUVFRQYROYHZLWKHLJKW LQSXWZLQGRZVWKHRXWSXWFRQVLVWVRIWZR2)0VDQGHDFKKDV ൈRXWSXWQHXURQVWKHQXPEHURISURFHVVLQJF\FOHV NLBs LV LQ)LJ D WRLOOXVWUDWHRXUGHVLJQ$IWHU0D[3OD\HU HDFK2)0RQO\KDVWZR(21VDVVKRZQLQMAX_Table,Q )LJ E GXHWRWKHWZRGLPHQVLRQVKDULQJWKLVUSPEDUUD\
ܲை ܰ௦ Data Loading:$IWHUWKHSUHGLFWLRQLQ3UHGLFWLRQ6WDJHZH REWDLQ WKH UDWLR RI (21V ߙ EHORQJLQJ WR WKH VDPH RXWSXW ORFDWLRQ7RNHHSDOOWKHUSPEVEXV\WKHHIIHFWXDOQXPEHURI ILOWHU ߙ ൈNf VKRXOGEHODUJHUWKDQWKHSDUDOOHOLVPRIILOWHUV PO ZKHUHNfLVWKHQXPEHURIILOWHUVZHQHHGWRVWRUHLQWKH JOREDOEXIIHU7KHUHIRUHNfFDQEHFDOFXODWHGDV
759
,GHDO
$FWXDO
2YHUDOOZR)&
2YHUDOO
&219
$OH[1HW 9** 9** 9**
1L1
6TX]1HW *HR
ܰ
ೀ ఈ
1L1
6TX]1HW
*HR
, WR SHUIRUP 3UHGLFWLRQ 6WDJH DQG WKH UHPDLQLQJ ELWV NHBs WRSHUIRUP([HFXWLRQ6WDJH 2XU EDVHOLQH LV WKH 0,)002)0 DUFKLWHFWXUH :H FKRRVH WKH VDPH SDUDOOHOL]DWLRQ FRQILJXUDWLRQ 3, DQG 32 IRURXUGHVLJQDQGWKHEDVHOLQH2XUUSPELVEDVHGRQ D VHULDO PXOWLSOLHU ZKLOH EDVHOLQH¶V SURFHVVLQJ HOHPHQW LV LPSOHPHQWHG ZLWK D ELW SDUDOOHO PXOWLSOLHU 7R PDLQWDLQ WKH FRPSDUDEOH SHUIRUPDQFH DV REWDLQHG E\ D SDUDOOHO PXOWLSOLHUZHVLPXOWDQHRXVO\SURFHVVRXWSXWQHXURQVLQ RQH 2)0 3/ :H LPSOHPHQW RXU GHVLJQ DQG EDVHOLQH XVLQJ;LOLQ[9&8(YDOXDWLRQERDUG,WLQFOXGHVWZRVHWV RI ILYH 0% ''5 6'5$0 DQG D ;LOLQ[ 8OWUD6FDOH ;&98 3)3*$
6LQFHWKHJOREDOEXIIHUFDQORDGPOFRQVHFXWLYHILOWHUVDW RQHWLPHLWQHHGVͳȀߙF\FOHVWRILQLVKWKHILOWHUORDGLQJ$V ߙLVQRUPDOO\ODUJHUWKDQDQGNLBsLVODUJHUWKDQWKH WLPHRIGDWDORDGLQJ ͳȀߙ LVOHVVWKDQWKDWRIGDWDSURFHVVLQJ ZKLFK GHPRQVWUDWHV LW FDQ PHHW WKH UHTXLUHPHQW RI GDWD ORDGLQJ C. Fully Connected Layer ,Q )&1 OD\HUV HDFK RXWSXW QHXURQ KDV HDFK RZQ ILOWHU 6LQFHHDFKUSPELVUHVSRQVLEOHIRURQHRXWSXWQHXURQHDFK USPEQHHGVWRORDGLWVRZQILOWHUDQGWKHUHLVQRILOWHUVKDULQJ LQWKHUSPEDUUD\7KHUHIRUHILOWHUDFFHVVLVDERWWOHQHFNIRU )&1 OD\HUV 7R NHHS DOO WKH USPEV EXV\ ZH QHHG WR ORDG PLൈPOILOWHUV,Q([HFXWLRQ6WDJHXQGHUWKHZRUVWFDVHWKH (21VDUHUDQGRPO\ORFDWHGLQWKH2)0ZKLFKPHDQVWKHLU FRUUHVSRQGLQJILOWHUVDUHDOVRUDQGRPO\ORFDWHGLQPHPRU\,W LV LPSRVVLEOH WR UDQGRPO\ ORDG PL ൈPO ILOWHUV ZLWKLQ NLBs F\FOHV7RDOOHYLDWHWKHSUHVVXUHRIPHPRU\DFFHVVZHDGG DQDFFXPXODWRUWRHDFKURZDQGPDNHWKHUSPEVLQRQHURZ UHVSRQVLEOHIRURQHRXWSXWQHXURQ(DFKWLPHZHORDGDILOWHU UHODWHGWR(21VDQGDVVLJQLWVZHLJKWVWRWKHUSPEVLQRQH URZ6LQFHWKHZHLJKWVEHORQJLQJWRDILOWHUDUHFRQVHFXWLYHO\ ORFDWHG LQ PHPRU\ ZH FDQ ORDG WKHP ZLWKLQ RQH F\FOH ,Q WKLVZD\LWWDNHVPOF\FOHVWRILQLVKWKHILOWHUDFFHVV$IWHU FKRRVLQJPOWKDWVDWLVILHV ZHFDQWDFNOHWKHFKDOOHQJHRI ILOWHUDFFHVVDQGPDNHWKHUSPEDUUD\EHLQJIXOO\XWLOL]HG 9
$OH[1HW 9** 9** 9**
)LJXUH2YHUDOO6SHHGXSDPRQJ'LIIHUHQW1HWZRUNV
)LJXUH6SHHGXSRI&219DPRQJ'LIIHUHQW1HWZRUNV
B. Performance Improvement )LJ LOOXVWUDWHV WKH VSHHGXS RI &219 OD\HUV DFURVV GLIIHUHQW'11VUHODWLYHWRWKHEDVHOLQH,Q)LJWKHLGHDO VSHHGXSLVFDOFXODWHGXQGHUWKHLGHDOFDVHZKHUHQRLGOHQHVV H[LVWV DQG WKH UHDO VSHHGXS LV DFKLHYHG E\ RXU KDUGZDUH LPSOHPHQWDWLRQ ,Q JHQHUDO RXU SUHGLFWLRQEDVHG GHVLJQ \LHOGV DQ LGHDO VSHHGXS RI ; DQG DQ DFWXDO VSHHGXS RI ; RQ DYHUDJH DFURVV DOO '11V RYHU WKH EDVHOLQH 7KLV LOOXVWUDWHV WKDW WKH DFWXDO VSHHGXS LV DSSURDFKLQJ WKH LGHDO VSHHGXSE\RXULGOHQHVVUHGXFWLRQGHVLJQLQ6HFWLRQ,9%,W DOVR VKRZV WKDW ZH H[SORLW WKH LQKHUHQW SUHGLFWDELOLW\ RI '11VWRUHPRYHQHDUO\DOOWKHPXOWLSOLFDWLRQVUHODWHGWRWKH L(21V )LJHYDOXDWHVWKHRYHUDOOSHUIRUPDQFHRIRXUGHVLJQ RQ ERWK FRQYROXWLRQ OD\HUV DQG )&1 OD\HUV :LWKRXW )&1 RSWLPL]DWLRQWKHRYHUDOOVSHHGXSZLOOGHJUDGHFRPSDUHGWR &219VSHHGXS:LWKRXURSWLPL]DWLRQIRU)&1OD\HUVWKH RYHUDOO VSHHGXS ZLOO EH LPSURYHG VOLJKWO\ IRU $OH[1HW DQG 9**)RURWKHUQHWZRUNVZKHUH)&1OD\HUVDFFRXQWVIRU YHU\ VPDOO SURSRUWLRQ RI WRWDO FRPSXWDWLRQV WKHLU SHUIRUPDQFHLVPDLQO\GHWHUPLQHGE\&219OD\HUVDQGWKH RYHUDOO VSHHGXS QHDUO\ NHHSV IL[HG $OWKRXJK RXU )&1 RSWLPL]DWLRQ KDV D OLPLWHG LPSURYHPHQW RQ SHUIRUPDQFH LW ZLOOJUHDWO\PLWLJDWHWKHPHPRU\DFFHVVRYHUKHDGDQGUHGXFH UHDVRQDEOHHQHUJ\ZKLFKZLOOEHLOOXVWUDWHGLQ6HFWLRQ9'
(9$/8$7,21
,QWKLVVHFWLRQZHILUVWLQWURGXFHRXUPHWKRGRORJ\7KHQ ZH HYDOXDWH WKH SHUIRUPDQFH LPSURYHPHQW DQG HQHUJ\ HIILFLHQF\ RI RXU GHVLJQ :H DOVR EUHDN GRZQ WKH SHUIRUPDQFHLPSURYHPHQWDQGH[SORUHWKHWUDGHRIIEHWZHHQ JOREDO EXIIHU VL]H DQG VSHHGXS )LQDOO\ ZH FRPSDUH RXU GHVLJQZLWKRWKHUVWDWHRIWKHDUWVFKHPHVVXFKDVCnvlutin >@DQG Stripes>@ :H IXUWKHU FRPELQH RXU GHVLJQ ZLWK Cnvlutin DQG Stirpes UHVSHFWLYHO\ WR GHPRQVWUDWH WKH LPSURYHPHQWLQWURGXFHGE\RXUPHWKRG
C. A Breakdown of Performance Improvement ,Q WKLV VHFWLRQ ZH LOOXVWUDWH WKH EUHDNGRZQ RI SHUIRUPDQFHLPSURYHPHQW&RPSDUHGZLWKWKHQDwYHGHVLJQ LQ6HFWLRQ,,&RXURSWLPL]DWLRQLQFOXGHVFRPSXWDWLRQUHXVH DQG DUFKLWHFWXUH UHXVH $V VKRZQ LQ )LJ DOWKRXJK WKH QDwYH GHVLJQ FDQ VNLS RYHU WKH L(21V LW RQO\ KDV ; VSHHGXSRQDYHUDJHZKLFKLVIDUIURPLGHDOVSHHGXS7KLVLV EHFDXVH WKH H[WUD RYHUKHDGV LQFOXGLQJ DUHD RYHUKHDG DQG
A. Methodology 7KHHYDOXDWLRQXVHVWKHVHWRISRSXODUDQGVWDWHRIWKH DUW FRQYROXWLRQDO QHXUDO QHWZRUNV >@ >@ >@ >@ >@ VKRZQ LQ 7$%/( , 7KH QHWZRUNV DUH SUHWUDLQHG XVLQJ 3\WRUFK>@7RNHHSWKHDFFXUDF\VDPHDVWKHEDVHOLQHZLWK ELWZHXVHWKHQXPEHURIKLJKRUGHUELWV NHBs LQ7$%/(
760
%XIIHU2YHUKHDG
6SHHGXSDW'LIIHUHQW1I
$OH[1HW
9**
9**
9**
1L1
6TX]1HW
%XIIHU2YHUKHDG .%
6SHHGXS
)LJXUH7UDGHRIIEHWZHHQ%XIIHU6L]HDQG6SHHGXS 1DLYH'HVLJQ
&RPS5HXVH
$UFK5HXVH
3UHG
Z2IIFKLSZR)&1 Z2IIFKLSZ)&1
ZR2IIFKLSZR)&1 ZR2IIFKLSZ)&1
(QHUJ\(IILFLHQF\
6SHHGXS
$OH[1HW 9** 9** 9** 1L1 6TX]1HW )LJXUH%UHDNGRZQRI3HUIRUPDQFH,PSURYHPHQW
*HR
FRPSXWDWLRQ RYHUKHDG LQWURGXFHG E\ SUHGLFWLRQ )RU VRPH QHWZRUNVVXFKDV1L1WKHRYHUKHDGVHYHQRIIVHWLWVEHQHILW &RPSDUHG ZLWK DUFKLWHFWXUH UHXVH FRPSXWDWLRQ UHXVH FRQWULEXWHV OLWWOH LPSURYHPHQW RQ SHUIRUPDQFH 7KLV LV EHFDXVH WKH EHQHILW RI FRPSXWDWLRQ UHXVH LV OLPLWHG E\ WKH QXPEHURIKLJKRUGHUELWVLQ3UHGLFWLRQ6WDJH7RUHGXFHWKH RYHUKHDGRISUHGLFWRUZHWU\WRPLQLPL]HWKHQXPEHURIELWV LQ 3UHGLFWLRQ 6WDJH ZKLFK ZLOO UHGXFH WKH EHQHILW RI FRPSXWDWLRQUHXVH ,QVWHDG RI GHGLFDWHG DUFKLWHFWXUHV IRU SUHGLFWRU DQG H[HFXWRURXUXQLIRUPDUFKLWHFWXUHLVVKDUHGE\SUHGLFWRUDQG H[HFXWRU DW GLIIHUHQW WLPH VORWV 7KLV DUFKLWHFWXUH UHXVH FDQ VLJQLILFDQWO\ LPSURYH WKH SHUIRUPDQFH E\ DYRLGLQJ WKH LGOHQHVVRIGHGLFDWHGDUFKLWHFWXUHVZKHUHRQHDUFKLWHFWXUHLV ZDLWLQJIRUWKHRWKHU
$OH[1HW 9** 9** 9** 1L1 6TX]1HW *HR )LJXUH(QHUJ\(IILFLHQF\&RPSDULVRQZ ZR)&12SWLPL]DWLRQ
ILOWHUVDQGWKHJURXSVL]HLVNf$VGLVFXVVHGLQ6HFWLRQ,9 %QRWRQO\LVWKHYDOXHRINfUHODWHGWRJOREDOEXIIHUVL]H EXWDOVRLWLQIOXHQFHVWKHLGOHQHVVRIUSPEDUUD\7KXVLQ)LJ ZH GHPRQVWUDWH WKH UHODWLRQVKLS EHWZHHQ JOREDO EXIIHU VL]HDQGVSHHGXSZLWKYDULRXVNf1RWHWKDWWKHJOREDOEXIIHU VL]HLVSURSRUWLRQDOWRNfVLQFHDOOWKHORDGHGILOWHUVQHHGWR EH VWRUHG LQ WKH JOREDO EXIIHU )RU WKH VSHHGXS LW ILUVW LQFUHDVHV WKHQ SODWHDXV DV Nf FRQWLQXHV WR LQFUHDVH $W WKH EHJLQQLQJVLQFHWKHNfLVVPDOOLWFRXOGQRWVDWLVI\ ZKLFK UHVXOWVLQWKHLGOHQHVVRIUSPEDUUD\:LWKLQFUHDVLQJNfWKH LGOHQHVVZLOOEHPLWLJDWHG$QGNfZLWKQHDUO\PDNHVWKLV USPEDUUD\EHLQJIXOO\XWLOL]HG7KXVDIWHUNfLVELJJHUWKDQ WKH VSHHGXS GRHV QRW FRQWLQXH WR LQFUHDVH )RU 6TXHH]H1HW WKH VSHHGXS HYHQ GHFUHDVHV ZKHQ Nf LV ODUJHU WKDQ7KLVLVPDLQO\FDXVHGE\LWVVPDOOQXPEHURIILOWHUV nF LQPRVWRIWKHOD\HUV,IWKHJURXSVL]H Nf LVELJJHUWKDQ nF VRPH RI USPEV EHFRPH LGOHQHVV GXH WR ODFN RI ILOWHUV ZKLFKUHVXOWVLQSHUIRUPDQFHGHJUDGDWLRQRI6TXHH]H1HW7R VXPXSWKHSHUIRUPDQFHLPSURYHPHQWFRPHVDWWKHH[SHQVH RIJOREDOEXIIHUVL]HDQGWKHSHUIRUPDQFHQRWDOZD\VLQFUHDVH :HVKRXOGFRQVLGHUWKLVWUDGHRIIGXULQJDFFHOHUDWRUGHVLJQ
D. Energy Efficiency ,QWKLVVHFWLRQZHFRPSDUHHQHUJ\HIILFLHQF\ZDQGZR )&1RSWLPL]DWLRQ$VVKRZQLQ)LJZLWKRXWFRQVLGHULQJ RIIFKLSGDWDDFFHVVRXUGHVLJQKDVDQDYHUDJH;HQHUJ\ HIILFLHQF\DQGWKHHQHUJ\HIILFLHQF\LVQHDUO\WKHVDPHHYHQ ZLWK)&1RSWLPL]DWLRQ7KLVLVEHFDXVHWKHFRPSXWDWLRQVLQ &219OD\HUVGRPLQDWHWKHHQHUJ\FRQVXPSWLRQZLWKRXWRII FKLSGDWDDFFHVV$QGRXUSUHGLFWLRQEDVHGGHVLJQFDQJUHDWO\ UHPRYHWKHFRPSXWDWLRQVLQ&219OD\HUVE\VNLSSLQJRYHU DOOWKHFRPSXWDWLRQVUHODWHGWRL(21V :KHQ FRQVLGHULQJ WKH RIIFKLS GDWD DFFHVV WKH )&1 RSWLPL]DWLRQ ZHLJKW VSOLWWLQJ SOD\V DQ LPSRUWDQW UROH LQ HQHUJ\ HIILFLHQF\ HVSHFLDOO\ IRU WKH QHWZRUNV ZLWK ODUJH DPRXQWRIILOWHUZHLJKWVLQ)&1OD\HUVVXFKDV$OH[1HWDQG 9**1HW 2XU )&1 RSWLPL]DWLRQ FDQ UHGXFH WKH RIIFKLS PHPRU\ DFFHVV E\ DYRLGLQJ ORDGLQJ WKH UHPDLQLQJ ELWV RI ILOWHU ZHLJKWV UHODWHG WR L(21V LQ ([HFXWLRQ 6WDJH DQG WKH GHFUHDVH LQ RIIFKLS PHPRU\ DFFHVV IXUWKHU FRQWULEXWHV WKH KLJK HQHUJ\HIILFLHQF\ $V VKRZQ LQ )LJ )&1 RSWLPL]DWLRQ LPSURYHV WKH DYHUDJH HQHUJ\HIILFLHQF\ IURP ;WR ;
F. Benefits for Other Works %\ OHYHUDJLQJ WKH LGHD RI SUHGLFWLRQ LQ '11V RXU DFFHOHUDWRU GHVLJQ DFKLHYHV SHUIRUPDQFH DQG HQHUJ\ HIILFLHQF\ LPSURYHPHQW 0RUH LPSRUWDQWO\ RXU SUHGLFWLRQ LGHD LV RUWKRJRQDO WR RWKHU VWDWHRIWKHDUW ZRUNV VXFK DV Cnvlutin DQG Stripes DQG FDQ SURYLGH WKH FRPPXQLW\ ZLWK PRUH RSSRUWXQLWLHV WR H[SORLW )LJ ILUVW FRPSDUH RXU GHVLJQ ZLWK Cnvlutin DQG Stripes WKHQ GHPRQVWUDWHV ZKDW ZLOOKDSSHQLIZHLQWHJUDWHWKHVHLGHDVWRJHWKHU7KHEDVHOLQH LVVWLOOWKH0,)202)0DUFKLWHFWXUH ,Q )LJ RXU GHVLJQ FDQ DFKLHYH ; VSHHGXS RQ DYHUDJHRYHUCnvlutin7KLVLVEHFDXVHRXUSUHGLFWLRQEDVHG GHVLJQFDQUHPRYHPRUHLQHIIHFWXDOPXOWLSOLFDWLRQVDVVKRZQ LQ)LJ'LIIHUHQWIURPH[SORLWLQJWKHVSDUVLW\RI'11VDV CnvlutinDQGRXUGHVLJQStripesOHYHUDJHVWKHYDULDQFHLQWKH QXPHULFDO SUHFLVLRQ UHTXLUHPHQWV RI '11V WR LPSURYH WKH SHUIRUPDQFH7KXVStripesQHHGVWRSHUIRUPWKHFDOFXODWLRQV
E. Trade-offs among Speedup, Idleness, and Overhead 'XHWRWKHOLPLWHGPHPRU\EDQGZLGWKZHFRXOGQRWORDG DOOWKHILOWHUV WRWDOQXPEHULVnF LQWRWKHJOREDOEXIIHUZLWKLQ NLBsF\FOHV7KHUHIRUHHDFKWLPHZHRQO\ORDGRQHJURXSRI
761
&19
6WULSHV
3UHGLFWLRQ
&193UHGLFWLRQ
6WULSHV3UHGLFWLRQ
6SHHGXS
$OH[1HW
1L1
9**
*HR
)LJXUH &RPSDULVRQVZLWKCnvlutin CNV DQGStripes
UHODWHGWRDOOWKHRXWSXWQHXURQVZKLOHRXUPHWKRGRQO\QHHGV WRSURFHVVWKH(21V&RPSDUHGZLWKStripesRXUPHWKRGKDV DQDYHUDJH;VSHHGXS 1H[W ZH FRPELQH RXU SUHGLFWLRQEDVHG GHVLJQ ZLWK Cnvlutin DQG Stripes UHVSHFWLYHO\ WR PHDVXUH WKH LPSURYHPHQW LQWURGXFHG E\ RXU PHWKRG 7KLV GHVLJQ FRPELQDWLRQ Prediction+Cnvlutin DFKLHYHDQDYHUDJH ; VSHHGXSRYHUCnvlutinEHFDXVHRXUGHVLJQFDQIXUWKHUUHPRYH QRQ]HURYDOXHG RSHUDQG PXOWLSOLFDWLRQ UHODWHG WR L(21V 6LQFH RXU FKDUDFWHUL]DWLRQ LQ 7$%/( , GHPRQVWUDWHV WKH SUHGLFWLRQ 3UHGLFWLRQ6WDJH QHHGVWKHOHVVSUHFLVLRQ NHBs WKDQWKDWRIStripes Nstripes RXUSUHGLFWLRQEDVHGPHWKRGFDQ EH XWLOL]HG LQ Stripes :LWK WKH FRPELQDWLRQ PHWKRG Prediction+Stripes LQ3UHGLFWLRQ6WDJHZHXVHNHBs-width LQSXWIRUSUHGLFWLRQ,Q([HFXWLRQ6WDJHZHFDQSHUIRUPWKH FDOFXODWLRQXVLQJWKHUHPDLQLQJELWV NStripesNHBs 6LQFHWKLV FRPELQDWLRQGHVLJQFDQVNLSRYHUWKHFDOFXODWLRQVUHODWHGWR L(21V LW KDV DQ DYHUDJH ; VSHHGXS RYHU Stripes 7KHUHIRUHWKHRXWSXWVSDUVLW\GLVFRYHUHGE\RXUSUHGLFWLRQ EDVHG PHWKRG LV D JRRG RSSRUWXQLW\ WR IXUWKHU LPSURYH WKH SHUIRUPDQFHRIRWKHUVWDWHRIWKHDUWZRUNV
RYHUDOOWKHFRPSXWDWLRQVUHODWHGWRWKHL(21VEIE>@FDQ DFFHOHUDWHWKHSURFHVVLQJRIWKHQHWZRUNVE\VNLSSLQJWKH]HUR LQSXWVDQGZHLJKWV+RZHYHULWOLPLWVWRWKHIXOO\FRQQHFWHG OD\HUVRID&11PRGHOZKLOHRXUZRUNFDQEHDSSOLHGWRERWK FRQYROXWLRQDOOD\HUV WKHPDMRULW\RIWKHFRPSXWDWLRQV DQG IXOO\ FRQQHFWHG OD\HUV Scalpel > @ SURSRVHV QRGHSUXQLQJ WHFKQLTXHWRUHPRYHUHGXQGDQWQRGHVLQ'11VE\XVLQJPDVN OD\HUVWRG\QDPLFDOO\LGHQWLI\DQGUHPRYHXQLPSRUWDQWQRGHV 1RWHWKDWWKHQRGHLQScalpelLQGLFDWHVDQRXWSXWIHDWXUHPDS ,IWKHQRGHLVXQLPSRUWDQWDOOWKHRXWSXWQHXURQVLQWKLVQRGH RXWSXW IHDWXUH PDS ZLOO EH UHPRYHG 2XU ZRUN RQO\ UHPRYHV WKH L(21V LQ HDFK RXWSXW IHDWXUH PDS 6LQFH WKH SUXQLQJLQScalpelLVPRUHDJJUHVVLYHLWQHHGVWRUHWUDLQWKH QHWZRUNWRUHVWRUHWKHDFFXUDF\ZKLOHRXUZRUNFDQDFKLHYH WKHVDPHDFFXUDF\DVWKHEDVHOLQHZLWKRXWUHWUDLQLQJ 9,, &21&/86,21 7KLV ZRUN ILUVW LQWURGXFHV WKH SUHGLFWLRQEDVHG LGHD WR '11DFFHOHUDWRUGHVLJQ%\SUHGLFWLQJWKHL(21VRI&219 DQG)&1LQDGYDQFHRXUDFFHOHUDWRUFDQVLJQLILFDQWO\HOLP LQDWHWKHLQHIIHFWXDOFRPSXWDWLRQDQGUHGXFHPHPRU\DFFHVV 7RPLQLPL]HWKHRYHUKHDGRISUHGLFWRUZHSURSRVHDXQLIRUP DUFKLWHFWXUHUSPEIRUSUHGLFWRUDQGH[HFXWRU2XUUSPEFDQ DOVR OHYHUDJH WKH FRPSXWDWLRQ UHXVH EHWZHHQ SUHGLFWRU DQG H[HFXWRUWRUHGXFHWKHFRPSXWDWLRQDORYHUKHDG7RLPSURYH WKH SURFHVVLQJ WKURXJKSXW ZH SUHVHQW VFDOHRXW GHVLJQ IRU USPE$QGRXUQRYHOVFDOHRXWGHVLJQFDQUHGXFHWKHLGOHQHVV RI USPE GXH WR WKH UDQGRPQHVV DQG VSDUVLW\ RI (21V (YDOXDWLRQUHVXOWVVKRZWKDWRXUSURSRVHGGHVLJQDFKLHYHVDQ DYHUDJH ; VSHHGXS DQG ; HQHUJ\HIILFLHQF\ RYHU WKH WUDGLWLRQDO DFFHOHUDWRU 2XU SUHGLFWLRQEDVHG GHVLJQ KDV DQ DYHUDJH;VSHHGXSRYHUCnvlutinDQG;VSHHGXSRYHU Stripes0RUHRYHUZKHQFRPELQHGZLWKRXUGHVLJQCnvlutin DQGStripesFDQEHLPSURYHGE\ ;DQG;RQDYHUDJH UHVSHFWLYHO\
9, 5(/$7(':25. 7KHFRPSXWDWLRQDOUHTXLUHPHQWVDQGDSSOLFDELOLW\RIGHHS QHXUDO QHWZRUNV KDYH SURPSWHG UHVHDUFKHUV WR GHVLJQ QXPHURXV SURSRVDOV IRU KDUGZDUH DFFHOHUDWLRQ ZLWK LPSOHPHQWDWLRQVRQHLWKHU)3*$V>@>@>@>@> @ >@ RU $6,&V >@±>@ >@±>@ >@ >@ >@±>@ DianNao >@ DQG DaDianNao >@ XWLOL]H D ODUJH JOREDO EXIIHU DV D VKDUHG VWRUDJH WR UHGXFH '5$0 DFFHVV HQHUJ\ FRQVXPSWLRQ )DUDEHW HW DO >@ SURSRVH D V\VWROLF DUFKLWHFWXUH FDOOHG NeuFlow ZKHUH ILOWHU ZHLJKWV UHPDLQ VWDWLRQDU\ LQ WKH UHJLVWHU WR PD[LPL]H WKHLU UHXVDELOLW\ 7R DFKLHYHKLJKHQHUJ\HIILFLHQF\Eyeriss>@SURSRVHVDURZ VWDWLRQDU\ GDWDIORZ E\ H[SORLWLQJ ORFDO GDWD UHXVH RI ILOWHU ZHLJKWVDQGLQSXWQHXURQV$OOWKHDERYHDFFHOHUDWRUVPDLQO\ IRFXV RQ UHGXFLQJ HQHUJ\ FRQVXPSWLRQ YLD PHPRU\ DFFHVV RSWLPL]DWLRQ%HVLGHVDFKLHYLQJKLJKHQHUJ\HIILFLHQF\RXU DFFHOHUDWRU FDQ VLJQLILFDQWO\ LPSURYH SHUIRUPDQFH ZLWK WKH QRYHOSUHGLFWLRQEDVHGWZRVWDJHFRPSXWLQJDUFKLWHFWXUH 7KH VSDUVLW\ RI '11V RSHQV D QHZ RSSRUWXQLW\ WR RSWLPL]HWKHSHUIRUPDQFHRIDFFHOHUDWRUV1HYHUWKHOHVVPRVW RI WKH H[LVWLQJ HIIRUWV HJ Cnvlutin DQG Eyeriss PDLQO\ IRFXVRQWKHVSDUVLW\LQILOWHUZHLJKWVDQGLQSXWIHDWXUHPDSV >@ >@ >@ >@ )RU H[DPSOH Eyeriss FDQ JDWH ]HUR LQSXWQHXURQFRPSXWDWLRQVWRIXUWKHUVDYHSRZHU,QVWHDGRI SRZHULQJRIIWKH]HURQHXURQFRPSXWDWLRQVCnvlutinGLUHFWO\ VNLSVRYHUWKH]HURLQSXWV'LIIHUHQWIURPCnvlutinZHVNLS
$&.12:/('*0(17 7KLV ZRUN LV VXSSRUWHG LQ SDUW E\ 16) JUDQWV &$5((5 E\ 65& JUDQWV +- 5-* E\ 0LFURVRIW 5HVHDUFK 7UXVWZRUWK\ &RPSXWLQJ 6DIH DQG 6FDODEOH 0XOWLFRUH &RPSXWLQJ$ZDUGVDQGE\WKUHH,%0)DFXOW\$ZDUGV 5()(5(1&(6 >@ $OH[ .UL]KHYVN\ ,O\D 6XWVNHYHU DQG *HRIIUH\ ( +LQWRQ ,PDJH1HW &ODVVLILFDWLRQZLWK'HHS&RQYROXWLRQDO1HXUDO1HWZRUNV,Q$GYDQFHV LQ1HXUDO,QIRUPDWLRQ3URFHVVLQJ6\VWHPV 1,36 >@ .DLPLQJ +H ;LDQJ\X =KDQJ 6KDRTLQJ 5HQ DQG -LDQ 6XQ 'HHS 5HVLGXDO /HDUQLQJ IRU ,PDJH 5HFRJQLWLRQ DU;LY 3UHSU DU;LYY >@ 6KDRTLQJ 5HQ .DLPLQJ +H 5RVV *LUVKLFN DQG -LDQ 6XQ )DVWHU 5 &11 7RZDUGV 5HDO7LPH 2EMHFW 'HWHFWLRQ ZLWK 5HJLRQ 3URSRVDO 1HWZRUNVDU;LY3UHSUDU;LY -XQ >@ $6LPRQ\DQ.DQG=LVVHUPDQ9HU\'HHS&RQYROXWLRQDO1HWZRUNV IRU/DUJH6FDOH,PDJH5HFRJQLWLRQ&R55YRODEV >@ 0LQJFRQJ 6RQJ @ 0LQJFRQJ 6RQJ @ 6KLMLQ=KDQJ=LGRQJ'X/HL=KDQJ+XL\LQJ/DQ6KDROL/LX/LQJ /L4L*XDQG@ $OL 6KDILHH $QLUEDQ 1DJ 1DYHHQ 0XUDOLPDQRKDU 5DMHHY %DODVXEUDPRQLDQ-RKQ3DXO6WUDFKDQ0LDR+X56WDQOH\:LOOLDPV DQG 9LYHN 6ULNXPDU ,6$$& $ &RQYROXWLRQDO 1HXUDO 1HWZRUN $FFHOHUDWRU ZLWK ,Q6LWX $QDORJ $ULWKPHWLF LQ &URVVEDUV ,Q $&0,((( UG $QQXDO ,QWHUQDWLRQDO 6\PSRVLXP RQ &RPSXWHU $UFKLWHFWXUH ,6&$ >@ @ ;LDQJ\X=KDQJ-LDQKXD=RX;LDQJ0LQJ.DLPLQJ+HDQG-LDQ6XQ (IILFLHQW DQG $FFXUDWH $SSUR[LPDWLRQV RI 1RQOLQHDU &RQYROXWLRQDO 1HWZRUNVDU;LY3UHSUDU;LY 1RY > @ 0LFKDHO )LJXUQRY 'PLWU\ 9HWURY DQG 3XVKPHHW .RKOL 3HUIRUDWHG&11V $FFHOHUDWLRQ WKURXJK (OLPLQDWLRQ RI 5HGXQGDQW &RQYROXWLRQVDU;LY3UHSUDU;LY$SU >@ &KHQJ 7DL 7RQJ ;LDR @ -RUJH $OEHULFLR 3DWULFN -XGG 7D\OHU +HWKHULQJWRQ 7RU $DPRGW 1DWDOLH(QULJKW-HUJHUDQG$QGUHDV0RVKRYRV&QYOXWLQ ,QHIIHFWXDO 1HXURQ)UHH 'HHS &RQYROXWLRQDO 1HXUDO 1HWZRUN &RPSXWLQJ ,Q $&0,((( UG $QQXDO ,QWHUQDWLRQDO 6\PSRVLXP RQ &RPSXWHU $UFKLWHFWXUH ,6&$
>@ &6Q &RQYROXWLRQDO 1HXUDO 1HWZRUNV IRU 9LVXDO 5HFRJQLWLRQ KWWS FVQJLWKXELRFRQYROXWLRQDOQHWZRUNV >@ 3DWULFN -XGG -RUJH $OEHULFLR 7D\OHU +HWKHULQJWRQ 7RU 0 $DPRGW DQG $QGUHDV 0RVKRYRV 6WULSHV %LW6HULDO 'HHS 1HXUDO 1HWZRUN &RPSXWLQJ,Q$&0,((( WK$QQXDO,QWHUQDWLRQDO6\PSRVLXPRQ 0LFURDUFKLWHFWXUH 0,&52 >@ 5HFWLILHU KWWSV HQZLNLSHGLDRUJZLNL5HFWLILHUB QHXUDOBQHWZRUNV >@ &KULVWLDQ 6]HJHG\ :HL /LX @ &RQYROXWLRQDO QHXUDO QHWZRUN KWWSV HQZLNLSHGLDRUJZLNL&RQYROXWLRQDOBQHXUDOBQHWZRUN >@ @ &KHQ=KDQJ3HQJ/L*XDQJ\X6XQ@ 0LQ/LQ4LDQJ&KHQDQG6KXLFKHQJ@ 0 6RQJ - =KDQJ + &KHQ DQG 7 /L 7RZDUGV (IILFLHQW 0LFURDUFKLWHFWXUDO'HVLJQIRU$FFHOHUDWLQJ8QVXSHUYLVHG*$1%DVHG 'HHS /HDUQLQJ ,Q ,((( ,QWHUQDWLRQDO 6\PSRVLXP RQ +LJK 3HUIRUPDQFH&RPSXWHU$UFKLWHFWXUH +3&$ >@ &OpPHQW )DUDEHW %HULQ 0DUWLQL %HQRLW &RUGD 3ROLQD $NVHOURG (XJHQLR &XOXUFLHOOR DQG @ -LHFDR
