HPMC User Guide v 1.00
© 2022 Bassem W. Jamaleddine
This chapter shows how to use your HPMC calculator to write openMP (OMP) programs to be run on the MIC. The programs are being offloaded to the MIC Co-Processor.
■ OFFLOADING TO MIC WITH Direct Access Programming Library
shm, shared memory transport (used for intra-node communication only). dapl, direct access programming library,
# OMP_NUM_THREADS=32 I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1u I_MPI_PERHOST=1 I_MPI_DEBUG=5 I_MPI_FABRICS=shm:dapl I_MPI_FALLBACK=0 /opt/INTEL-XE-2017-update7/compilers_and_libraries_2017.7.259/linux/mpi/intel64/bin/mpiexec.hydra -n 10 -hosts hpmc9 ./trap_offload
02:36 root@HPMC9: /mm03fs/MPI # OMP_NUM_THREADS=32 I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1u I_MPI_PERHOST=1 I_MPI_DEBUG=5 I_MPI_FABRICS=shm:dapl I_MPI_FALLBACK=0 /opt/INTEL-XE-2017-update7/compilers_and_libraries_2017.7.259/linux/mpi/intel64/bin/mpiexec.hydra -n 10 -hosts hpmc9 ./trap_offload
[0] MPI startup(): Multi-threaded optimized library
[3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[7] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[8] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[6] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[9] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[4] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[4] MPI startup(): shm and dapl data transfer modes
[2] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[3] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[2] MPI startup(): shm and dapl data transfer modes
[3] MPI startup(): shm and dapl data transfer modes
[1] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[1] MPI startup(): shm and dapl data transfer modes
[7] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[8] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[8] MPI startup(): shm and dapl data transfer modes
[7] MPI startup(): shm and dapl data transfer modes
[6] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[9] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[5] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[6] MPI startup(): shm and dapl data transfer modes
[9] MPI startup(): shm and dapl data transfer modes
[0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[5] MPI startup(): shm and dapl data transfer modes
[0] MPI startup(): shm and dapl data transfer modes
[0] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[0] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[1] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[1] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[2] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[2] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[3] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[3] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[4] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[4] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[5] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[5] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[6] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[6] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[7] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[7] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[8] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[8] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[9] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[9] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 42442 HPMC9 {0,1,2,3,4,5,56,57,58,59,60}
[0] MPI startup(): 1 42443 HPMC9 {6,7,8,9,10,61,62,63,64,65,66}
[0] MPI startup(): 2 42444 HPMC9 {11,12,13,14,15,16,67,68,69,70,71}
[0] MPI startup(): 3 42445 HPMC9 {17,18,19,20,21,72,73,74,75,76,77}
[0] MPI startup(): 4 42446 HPMC9 {22,23,24,25,26,27,78,79,80,81,82}
[0] MPI startup(): 5 42447 HPMC9 {28,29,30,31,32,83,84,85,86,87,88}
[0] MPI startup(): 6 42448 HPMC9 {33,34,35,36,37,38,89,90,91,92,93}
[0] MPI startup(): 7 42449 HPMC9 {39,40,41,42,43,94,95,96,97,98,99}
[0] MPI startup(): 8 42450 HPMC9 {44,45,46,47,48,49,100,101,102,103,104}
[0] MPI startup(): 9 42451 HPMC9 {50,51,52,53,54,105,106,107,108,109,110}
[0] MPI startup(): I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1u
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_FABRICS=shm:dapl
[0] MPI startup(): I_MPI_FALLBACK=0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=i40iw0:0,i40iw1:0,qib0:0,qib1:0,mic0:1
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=2
[0] MPI startup(): I_MPI_PIN_MAPPING=10:0 0,1 6,2 11,3 17,4 22,5 28,6 33,7 39,8 44,9 50
rank 0 of 10 on HPMC9: 40.471704 seconds
integral = 1.856399, time = 45.410044
rank 1 of 10 on HPMC9: 41.719513 seconds
rank 2 of 10 on HPMC9: 43.035928 seconds
rank 3 of 10 on HPMC9: 44.055897 seconds
rank 4 of 10 on HPMC9: 44.575862 seconds
rank 5 of 10 on HPMC9: 45.216055 seconds
rank 6 of 10 on HPMC9: 45.299586 seconds
rank 7 of 10 on HPMC9: 44.915891 seconds
rank 8 of 10 on HPMC9: 45.410034 seconds
rank 9 of 10 on HPMC9: 45.268067 seconds
On the MIC we can reveal what is being run by the offloaded program.
(HPMC9) 01:32 root@mic0: ~ # pstree -up
init(1)-+-coi_daemon(4354,micuser)-+-trap_offloadMIC(5004)-+-{trap_offloadMIC}(5009)
| | |-{trap_offloadMIC}(5011)
| | |-{trap_offloadMIC}(5012)
| | |-{trap_offloadMIC}(5014)
| | |-{trap_offloadMIC}(5015)
| | |-{trap_offloadMIC}(5017)
| | |-{trap_offloadMIC}(5018)
| | |-{trap_offloadMIC}(5019)
| | |-{trap_offloadMIC}(5020)
| | |-{trap_offloadMIC}(5021)
| | |-{trap_offloadMIC}(5022)
| | |-{trap_offloadMIC}(5023)
| | |-{trap_offloadMIC}(5024)
| | |-{trap_offloadMIC}(5025)
| | |-{trap_offloadMIC}(5026)
| | |-{trap_offloadMIC}(5027)
| | |-{trap_offloadMIC}(5028)
| | |-{trap_offloadMIC}(5029)
| | |-{trap_offloadMIC}(5030)
| | |-{trap_offloadMIC}(5031)
| | |-{trap_offloadMIC}(5032)
| | |-{trap_offloadMIC}(5033)
| | |-{trap_offloadMIC}(5034)
| | |-{trap_offloadMIC}(5035)
| | |-{trap_offloadMIC}(5036)
| | |-{trap_offloadMIC}(5037)
| | |-{trap_offloadMIC}(5038)
| | |-{trap_offloadMIC}(5039)
| | |-{trap_offloadMIC}(5040)
| | |-{trap_offloadMIC}(5041)
| | |-{trap_offloadMIC}(5042)
| | |-{trap_offloadMIC}(5043)
| | `-{trap_offloadMIC}(5044)
| |-trap_offloadMIC(5013)-+-{trap_offloadMIC}(5047)
| | |-{trap_offloadMIC}(5049)
| | |-{trap_offloadMIC}(5052)
| | |-{trap_offloadMIC}(5053)
| | |-{trap_offloadMIC}(5054)
| | |-{trap_offloadMIC}(5055)
| | |-{trap_offloadMIC}(5056)
| | |-{trap_offloadMIC}(5057)
| | |-{trap_offloadMIC}(5058)
| | |-{trap_offloadMIC}(5059)
| | |-{trap_offloadMIC}(5060)
| | |-{trap_offloadMIC}(5061)
| | |-{trap_offloadMIC}(5062)
| | |-{trap_offloadMIC}(5063)
| | |-{trap_offloadMIC}(5064)
| | |-{trap_offloadMIC}(5065)
| | |-{trap_offloadMIC}(5066)
| | |-{trap_offloadMIC}(5067)
| | |-{trap_offloadMIC}(5068)
| | |-{trap_offloadMIC}(5069)
| | |-{trap_offloadMIC}(5070)
| | |-{trap_offloadMIC}(5071)
| | |-{trap_offloadMIC}(5072)
| | |-{trap_offloadMIC}(5074)
| | |-{trap_offloadMIC}(5075)
| | |-{trap_offloadMIC}(5076)
| | |-{trap_offloadMIC}(5077)
| | |-{trap_offloadMIC}(5078)
| | |-{trap_offloadMIC}(5079)
| | |-{trap_offloadMIC}(5080)
| | |-{trap_offloadMIC}(5081)
| | |-{trap_offloadMIC}(5082)
| | `-{trap_offloadMIC}(5083)
| |-trap_offloadMIC(5050)-+-{trap_offloadMIC}(5085)
| | |-{trap_offloadMIC}(5087)
| | |-{trap_offloadMIC}(5088)
| | |-{trap_offloadMIC}(5089)
| | |-{trap_offloadMIC}(5090)
| | |-{trap_offloadMIC}(5091)
| | |-{trap_offloadMIC}(5092)
| | |-{trap_offloadMIC}(5094)
| | |-{trap_offloadMIC}(5096)
| | |-{trap_offloadMIC}(5097)
| | |-{trap_offloadMIC}(5099)
| | |-{trap_offloadMIC}(5100)
| | |-{trap_offloadMIC}(5101)
| | |-{trap_offloadMIC}(5102)
| | |-{trap_offloadMIC}(5103)
| | |-{trap_offloadMIC}(5104)
| | |-{trap_offloadMIC}(5105)
| | |-{trap_offloadMIC}(5106)
| | |-{trap_offloadMIC}(5107)
| | |-{trap_offloadMIC}(5108)
| | |-{trap_offloadMIC}(5111)
| | |-{trap_offloadMIC}(5112)
| | |-{trap_offloadMIC}(5115)
| | |-{trap_offloadMIC}(5116)
| | |-{trap_offloadMIC}(5117)
| | |-{trap_offloadMIC}(5120)
| | |-{trap_offloadMIC}(5123)
| | |-{trap_offloadMIC}(5126)
| | |-{trap_offloadMIC}(5128)
| | |-{trap_offloadMIC}(5130)
| | |-{trap_offloadMIC}(5132)
| | |-{trap_offloadMIC}(5134)
| | `-{trap_offloadMIC}(5135)
| |-trap_offloadMIC(5093)-+-{trap_offloadMIC}(5110)
| | |-{trap_offloadMIC}(5114)
| | |-{trap_offloadMIC}(5118)
| | |-{trap_offloadMIC}(5119)
| | |-{trap_offloadMIC}(5121)
| | |-{trap_offloadMIC}(5122)
| | |-{trap_offloadMIC}(5125)
| | |-{trap_offloadMIC}(5129)
| | |-{trap_offloadMIC}(5131)
| | |-{trap_offloadMIC}(5133)
| | |-{trap_offloadMIC}(5137)
| | |-{trap_offloadMIC}(5138)
| | |-{trap_offloadMIC}(5139)
| | |-{trap_offloadMIC}(5140)
| | |-{trap_offloadMIC}(5141)
| | |-{trap_offloadMIC}(5143)
| | |-{trap_offloadMIC}(5147)
| | |-{trap_offloadMIC}(5150)
| | |-{trap_offloadMIC}(5153)
| | |-{trap_offloadMIC}(5155)
| | |-{trap_offloadMIC}(5157)
| | |-{trap_offloadMIC}(5161)
| | |-{trap_offloadMIC}(5163)
| | |-{trap_offloadMIC}(5165)
| | |-{trap_offloadMIC}(5166)
| | |-{trap_offloadMIC}(5168)
| | |-{trap_offloadMIC}(5169)
| | |-{trap_offloadMIC}(5172)
| | |-{trap_offloadMIC}(5176)
| | |-{trap_offloadMIC}(5177)
| | |-{trap_offloadMIC}(5181)
| | |-{trap_offloadMIC}(5183)
| | `-{trap_offloadMIC}(5186)
| |-trap_offloadMIC(5124)-+-{trap_offloadMIC}(5144)
| | |-{trap_offloadMIC}(5146)
| | |-{trap_offloadMIC}(5148)
| | |-{trap_offloadMIC}(5152)
| | |-{trap_offloadMIC}(5154)
| | |-{trap_offloadMIC}(5156)
| | |-{trap_offloadMIC}(5158)
| | |-{trap_offloadMIC}(5160)
| | |-{trap_offloadMIC}(5162)
| | |-{trap_offloadMIC}(5164)
| | |-{trap_offloadMIC}(5167)
| | |-{trap_offloadMIC}(5171)
| | |-{trap_offloadMIC}(5174)
| | |-{trap_offloadMIC}(5175)
| | |-{trap_offloadMIC}(5179)
| | |-{trap_offloadMIC}(5182)
| | |-{trap_offloadMIC}(5185)
| | |-{trap_offloadMIC}(5189)
| | |-{trap_offloadMIC}(5191)
| | |-{trap_offloadMIC}(5192)
| | |-{trap_offloadMIC}(5195)
| | |-{trap_offloadMIC}(5197)
| | |-{trap_offloadMIC}(5199)
| | |-{trap_offloadMIC}(5201)
| | |-{trap_offloadMIC}(5203)
| | |-{trap_offloadMIC}(5205)
| | |-{trap_offloadMIC}(5210)
| | |-{trap_offloadMIC}(5215)
| | |-{trap_offloadMIC}(5218)
| | |-{trap_offloadMIC}(5223)
| | |-{trap_offloadMIC}(5225)
| | |-{trap_offloadMIC}(5228)
| | `-{trap_offloadMIC}(5231)
| |-trap_offloadMIC(5149)-+-{trap_offloadMIC}(5173)
| | |-{trap_offloadMIC}(5180)
| | |-{trap_offloadMIC}(5188)
| | |-{trap_offloadMIC}(5190)
| | |-{trap_offloadMIC}(5193)
| | |-{trap_offloadMIC}(5196)
| | |-{trap_offloadMIC}(5198)
| | |-{trap_offloadMIC}(5200)
| | |-{trap_offloadMIC}(5202)
| | |-{trap_offloadMIC}(5204)
| | |-{trap_offloadMIC}(5208)
| | |-{trap_offloadMIC}(5209)
| | |-{trap_offloadMIC}(5216)
| | |-{trap_offloadMIC}(5217)
| | |-{trap_offloadMIC}(5221)
| | |-{trap_offloadMIC}(5222)
| | |-{trap_offloadMIC}(5227)
| | |-{trap_offloadMIC}(5229)
| | |-{trap_offloadMIC}(5232)
| | |-{trap_offloadMIC}(5233)
| | |-{trap_offloadMIC}(5236)
| | |-{trap_offloadMIC}(5237)
| | |-{trap_offloadMIC}(5238)
| | |-{trap_offloadMIC}(5243)
| | |-{trap_offloadMIC}(5244)
| | |-{trap_offloadMIC}(5250)
| | |-{trap_offloadMIC}(5252)
| | |-{trap_offloadMIC}(5257)
| | |-{trap_offloadMIC}(5258)
| | |-{trap_offloadMIC}(5262)
| | |-{trap_offloadMIC}(5263)
| | |-{trap_offloadMIC}(5269)
| | `-{trap_offloadMIC}(5274)
| |-trap_offloadMIC(5184)-+-{trap_offloadMIC}(5207)
| | |-{trap_offloadMIC}(5213)
| | |-{trap_offloadMIC}(5219)
| | |-{trap_offloadMIC}(5224)
| | |-{trap_offloadMIC}(5226)
| | |-{trap_offloadMIC}(5230)
| | |-{trap_offloadMIC}(5234)
| | |-{trap_offloadMIC}(5235)
| | |-{trap_offloadMIC}(5239)
| | |-{trap_offloadMIC}(5242)
| | |-{trap_offloadMIC}(5249)
| | |-{trap_offloadMIC}(5254)
| | |-{trap_offloadMIC}(5259)
| | |-{trap_offloadMIC}(5265)
| | |-{trap_offloadMIC}(5268)
| | |-{trap_offloadMIC}(5271)
| | |-{trap_offloadMIC}(5276)
| | |-{trap_offloadMIC}(5279)
| | |-{trap_offloadMIC}(5282)
| | |-{trap_offloadMIC}(5284)
| | |-{trap_offloadMIC}(5286)
| | |-{trap_offloadMIC}(5292)
| | |-{trap_offloadMIC}(5294)
| | |-{trap_offloadMIC}(5300)
| | |-{trap_offloadMIC}(5305)
| | |-{trap_offloadMIC}(5306)
| | |-{trap_offloadMIC}(5314)
| | |-{trap_offloadMIC}(5317)
| | |-{trap_offloadMIC}(5322)
| | |-{trap_offloadMIC}(5326)
| | |-{trap_offloadMIC}(5331)
| | |-{trap_offloadMIC}(5336)
| | `-{trap_offloadMIC}(5339)
| |-trap_offloadMIC(5212)-+-{trap_offloadMIC}(5241)
| | |-{trap_offloadMIC}(5248)
| | |-{trap_offloadMIC}(5253)
| | |-{trap_offloadMIC}(5255)
| | |-{trap_offloadMIC}(5256)
| | |-{trap_offloadMIC}(5260)
| | |-{trap_offloadMIC}(5261)
| | |-{trap_offloadMIC}(5264)
| | |-{trap_offloadMIC}(5267)
| | |-{trap_offloadMIC}(5273)
| | |-{trap_offloadMIC}(5277)
| | |-{trap_offloadMIC}(5283)
| | |-{trap_offloadMIC}(5287)
| | |-{trap_offloadMIC}(5289)
| | |-{trap_offloadMIC}(5293)
| | |-{trap_offloadMIC}(5297)
| | |-{trap_offloadMIC}(5302)
| | |-{trap_offloadMIC}(5304)
| | |-{trap_offloadMIC}(5307)
| | |-{trap_offloadMIC}(5308)
| | |-{trap_offloadMIC}(5309)
| | |-{trap_offloadMIC}(5310)
| | |-{trap_offloadMIC}(5311)
| | |-{trap_offloadMIC}(5316)
| | |-{trap_offloadMIC}(5319)
| | |-{trap_offloadMIC}(5328)
| | |-{trap_offloadMIC}(5333)
| | |-{trap_offloadMIC}(5337)
| | |-{trap_offloadMIC}(5343)
| | |-{trap_offloadMIC}(5346)
| | |-{trap_offloadMIC}(5352)
| | |-{trap_offloadMIC}(5356)
| | `-{trap_offloadMIC}(5357)
| |-trap_offloadMIC(5246)-+-{trap_offloadMIC}(5270)
| | |-{trap_offloadMIC}(5280)
| | |-{trap_offloadMIC}(5285)
| | |-{trap_offloadMIC}(5288)
| | |-{trap_offloadMIC}(5290)
| | |-{trap_offloadMIC}(5291)
| | |-{trap_offloadMIC}(5295)
| | |-{trap_offloadMIC}(5296)
| | |-{trap_offloadMIC}(5298)
| | |-{trap_offloadMIC}(5299)
| | |-{trap_offloadMIC}(5313)
| | |-{trap_offloadMIC}(5318)
| | |-{trap_offloadMIC}(5321)
| | |-{trap_offloadMIC}(5323)
| | |-{trap_offloadMIC}(5327)
| | |-{trap_offloadMIC}(5330)
| | |-{trap_offloadMIC}(5332)
| | |-{trap_offloadMIC}(5335)
| | |-{trap_offloadMIC}(5340)
| | |-{trap_offloadMIC}(5344)
| | |-{trap_offloadMIC}(5347)
| | |-{trap_offloadMIC}(5349)
| | |-{trap_offloadMIC}(5353)
| | |-{trap_offloadMIC}(5358)
| | |-{trap_offloadMIC}(5363)
| | |-{trap_offloadMIC}(5365)
| | |-{trap_offloadMIC}(5367)
| | |-{trap_offloadMIC}(5368)
| | |-{trap_offloadMIC}(5370)
| | |-{trap_offloadMIC}(5371)
| | |-{trap_offloadMIC}(5374)
| | |-{trap_offloadMIC}(5375)
| | `-{trap_offloadMIC}(5377)
| |-trap_offloadMIC(5272)-+-{trap_offloadMIC}(5303)
| | |-{trap_offloadMIC}(5315)
| | |-{trap_offloadMIC}(5320)
| | |-{trap_offloadMIC}(5324)
| | |-{trap_offloadMIC}(5325)
| | |-{trap_offloadMIC}(5329)
| | |-{trap_offloadMIC}(5334)
| | |-{trap_offloadMIC}(5338)
| | |-{trap_offloadMIC}(5341)
| | |-{trap_offloadMIC}(5342)
| | |-{trap_offloadMIC}(5345)
| | |-{trap_offloadMIC}(5348)
| | |-{trap_offloadMIC}(5350)
| | |-{trap_offloadMIC}(5351)
| | |-{trap_offloadMIC}(5354)
| | |-{trap_offloadMIC}(5355)
| | |-{trap_offloadMIC}(5359)
| | |-{trap_offloadMIC}(5360)
| | |-{trap_offloadMIC}(5361)
| | |-{trap_offloadMIC}(5362)
| | |-{trap_offloadMIC}(5364)
| | |-{trap_offloadMIC}(5366)
| | |-{trap_offloadMIC}(5369)
| | |-{trap_offloadMIC}(5372)
| | |-{trap_offloadMIC}(5373)
| | |-{trap_offloadMIC}(5376)
| | |-{trap_offloadMIC}(5378)
| | |-{trap_offloadMIC}(5379)
| | |-{trap_offloadMIC}(5380)
| | |-{trap_offloadMIC}(5381)
| | |-{trap_offloadMIC}(5382)
| | |-{trap_offloadMIC}(5383)
| | `-{trap_offloadMIC}(5384)
| |-{coi_daemon}(4356)
| `-{coi_daemon}(5006)
|-getty(4370)
|-klogd(4333)
|-mpssd(4339)-+-{mpssd}(4341)
| `-{mpssd}(4342)
|-portmap(4290,1)
|-sshd(4322)---sshd(4372)---bash(4374)---pstree(5385)
|-syslogd(4330)
`-udevd(4113)-+-udevd(4336)
`-udevd(4363)
■ MPI OFFLOADING TO TWO MICs
Testing with the mpiexec.hydra by offloading to two MICs on the hosts: hpmc9 and hpmc7.
# OMP_NUM_THREADS=32 I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1u I_MPI_PERHOST=1 I_MPI_DEBUG=5 I_MPI_FABRICS=shm:dapl I_MPI_FALLBACK=0 /opt/INTEL-XE-2017-update7/compilers_and_libraries_2017.7.259/linux/mpi/intel64/bin/mpiexec.hydra -n 10 -hosts hpmc9,hpmc7 ./trap_offload
[0] MPI startup(): Multi-threaded optimized library
[0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[6] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[8] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[7] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[9] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx4_0-1u
[0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[0] MPI startup(): shm and dapl data transfer modes
[2] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[2] MPI startup(): shm and dapl data transfer modes
[4] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[4] MPI startup(): shm and dapl data transfer modes
[8] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[6] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[6] MPI startup(): shm and dapl data transfer modes
[8] MPI startup(): shm and dapl data transfer modes
[1] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[3] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[5] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[7] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[1] MPI startup(): shm and dapl data transfer modes
[3] MPI startup(): shm and dapl data transfer modes
[5] MPI startup(): shm and dapl data transfer modes
[7] MPI startup(): shm and dapl data transfer modes
[9] MPI startup(): DAPL provider ofa-v2-mlx4_0-1u
[9] MPI startup(): shm and dapl data transfer modes
[0] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[0] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[2] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[2] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[4] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[4] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[6] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[6] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[8] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[8] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[1] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[1] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[3] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[3] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[5] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[5] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[7] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[7] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[9] MPID_nem_init_dapl_coll_fns(): User set DAPL collective mask = 0000
[9] MPID_nem_init_dapl_coll_fns(): Effective DAPL collective mask = 0000
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 42103 HPMC9 {0,1,2,3,4,5,6,7,8,9,10,56,57,58,59,60,61,62,63,64,65,66}
[0] MPI startup(): 1 32814 HPMC7 {0,1,2,3,4,5,6,7,8,9,10,11,12,13,68,69,70,71,72,73,74,75,76,77,78,79,80,81,136,13
7,138,139,140,141,142,143,144,145,146,147,148,204,205,206,207,208,209,210,211,21
2,213,214,215,216}
[0] MPI startup(): 2 42104 HPMC9 {11,12,13,14,15,16,17,18,19,20,21,67,68,69,70,71,72,73,74,75,76,77}
[0] MPI startup(): 3 32815 HPMC7 {14,15,16,17,18,19,20,21,22,23,24,25,26,82,83,84,85,86,87,88,89,90,91,92,93,94,14
9,150,151,152,153,154,155,156,157,158,159,160,161,162,217,218,219,220,221,222,22
3,224,225,226,227,228,229,230}
[0] MPI startup(): 4 42105 HPMC9 {22,23,24,25,26,27,28,29,30,31,32,78,79,80,81,82,83,84,85,86,87,88}
[0] MPI startup(): 5 32816 HPMC7 {27,28,29,30,31,32,33,34,35,36,37,38,39,40,95,96,97,98,99,100,101,102,103,104,105
,106,107,108,163,164,165,166,167,168,169,170,171,172,173,174,175,231,232,233,234
,235,236,237,238,239,240,241,242,243}
[0] MPI startup(): 6 42106 HPMC9 {33,34,35,36,37,38,39,40,41,42,43,89,90,91,92,93,94,95,96,97,98,99}
[0] MPI startup(): 7 32817 HPMC7 {41,42,43,44,45,46,47,48,49,50,51,52,53,109,110,111,112,113,114,115,116,117,118,1
19,120,121,176,177,178,179,180,181,182,183,184,185,186,187,188,189,244,245,246,2
47,248,249,250,251,252,253,254,255,256,257}
[0] MPI startup(): 8 42107 HPMC9 {44,45,46,47,48,49,50,51,52,53,54,100,101,102,103,104,105,106,107,108,109,110}
[0] MPI startup(): 9 32818 HPMC7 {54,55,56,57,58,59,60,61,62,63,64,65,66,67,122,123,124,125,126,127,128,129,130,13
1,132,133,134,135,190,191,192,193,194,195,196,197,198,199,200,201,202,258,259,26
0,261,262,263,264,265,266,267,268,269,270}
[0] MPI startup(): I_MPI_DAPL_PROVIDER_LIST=ofa-v2-mlx4_0-1u
[0] MPI startup(): I_MPI_DEBUG=5
[0] MPI startup(): I_MPI_FABRICS=shm:dapl
[0] MPI startup(): I_MPI_FALLBACK=0
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=i40iw0:0,i40iw1:0,qib0:0,qib1:0,mic0:1
[0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=2
[0] MPI startup(): I_MPI_PIN_MAPPING=5:0 0,2 11,4 22,6 33,8 44
rank 4 of 10 on HPMC9: 22.456818 seconds
rank 1 of 10 on HPMC7: 20.294824 seconds
rank 6 of 10 on HPMC9: 22.954011 seconds
rank 3 of 10 on HPMC7: 18.985285 seconds
rank 8 of 10 on HPMC9: 22.732662 seconds
rank 5 of 10 on HPMC7: 19.954661 seconds
rank 0 of 10 on HPMC9: 21.098834 seconds
integral = 1.856399, time = 22.954172
rank 7 of 10 on HPMC7: 19.414617 seconds
rank 2 of 10 on HPMC9: 21.718827 seconds
rank 9 of 10 on HPMC7: 20.453557 seconds