Segmentaion fault while running on cluster

[umamahesh999]

Got below error while running wordembedding on cluster.

I am using starcluster to create cluster and mpi to run the jobs.

[node001:04462] *** Process received signal ***
[node001:04462] Signal: Segmentation fault (11)
[node001:04462] Signal code: Invalid permissions (2)
[node001:04462] Failing at address: 0x7f5232168010
[node001:04462] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f5271b09390]
[node001:04462] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x9f840)[0x7f52717cd840]
[node001:04462] [ 2] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso7UpdaterIfE6AccessEmPfS2_mPNS_9AddOptionE+0x4e)[0x7f527290e050]
[node001:04462] [ 3] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZN10multiverso17MatrixServerTableIfE10ProcessGetERKSt6vectorINS_4BlobESaIS3_EEPS5+0x39e)[0x7f52728f81a0]
[node001:04462] [ 4] [node004:04351] *** Process received signal ***
/home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso6Server10ProcessGetERSt10unique_ptrINS_7MessageESt14default_deleteIS2_EE+0x1df)[0x7f52728e132d]
[node001:04462] [ 5] [node004:04351] Signal: Segmentation fault (11)
[node004:04351] Signal code: Invalid permissions (2)
[node004:04351] Failing at address: 0x7fce92be0810
/home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso6ServerEFvRSt10unique_ptrINS0_7MessageESt14default_deleteIS3_EEELb1EEclIIS7_EvEEvPS1_DpOT+0x7c)[0x7f52728e61c2]
[node001:04462] [ 6] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEE6__callIvJS8_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE+0x84)[0x7f52728e5bb4]
[node001:04462] [ 7] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEEclIJS8_EvEET0_DpOT+0x56)[0x7f52728e4dce]
[node001:04462] [ 8] [node004:04351] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt17_Function_handlerIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEESt5_BindIFSt7_Mem_fnIMNS1_6ServerEFvS6_EEPSA_St12_PlaceholderILi1EEEEE9_M_invokeERKSt9_Any_dataS6+0x37)[0x7f52728e3f55]
[node001:04462] [ 9] [ 0] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt8functionIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEEEclES6+0x49)[0x7f52728c3d8f]
[node001:04462] [10] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso5Actor4MainEv+0xf3)[0x7f52728c2153]
[node001:04462] [11] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fced2544390]
/home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso5ActorEFvvELb1EEclIJEvEEvPS1_DpOT+0x65)[0x7f52728c90e3]
[node001:04462] [12] [node004:04351] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EE9_M_invokeIILm0EEEEvSt12_Index_tupleIIXspT_EEE+0x43)[0x7f52728c9077]
[node001:04462] [13] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EEclEv+0x2c)[0x7f52728c8f7e]
[node001:04462] [14] [ 1] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS4_EEE6_M_runEv+0x1c)[0x7f52728c8f0e]
[node001:04462] [15] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7f527250ec80]
[node001:04462] [16] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f5271aff6ba]
[node001:04462] [17] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f52718353dd]
[node001:04462] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x9f840)[0x7fced2208840]
[node004:04351] [ 2] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso7UpdaterIfE6AccessEmPfS2_mPNS_9AddOptionE+0x4e)[0x7fced3349050]
[node004:04351] [ 3] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZN10multiverso17MatrixServerTableIfE10ProcessGetERKSt6vectorINS_4BlobESaIS3_EEPS5+0x39e)[0x7fced33331a0]
[node004:04351] [ 4] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso6Server10ProcessGetERSt10unique_ptrINS_7MessageESt14default_deleteIS2_EE+0x1df)[0x7fced331c32d]
[node004:04351] [ 5] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso6ServerEFvRSt10unique_ptrINS0_7MessageESt14default_deleteIS3_EEELb1EEclIIS7_EvEEvPS1_DpOT+0x7c)[0x7fced33211c2]
[node004:04351] [ 6] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEE6__callIvJS8_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE+0x84)[0x7fced3320bb4]
[node004:04351] [ 7] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEEclIJS8_EvEET0_DpOT+0x56)[0x7fced331fdce]
[node004:04351] [ 8] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt17_Function_handlerIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEESt5_BindIFSt7_Mem_fnIMNS1_6ServerEFvS6_EEPSA_St12_PlaceholderILi1EEEEE9_M_invokeERKSt9_Any_dataS6+0x37)[0x7fced331ef55]
[node004:04351] [ 9] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt8functionIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEEEclES6+0x49)[0x7fced32fed8f]
[node004:04351] [10] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso5Actor4MainEv+0xf3)[0x7fced32fd153]
[node004:04351] [11] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso5ActorEFvvELb1EEclIJEvEEvPS1_DpOT+0x65)[0x7fced33040e3]
[node004:04351] [12] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EE9_M_invokeIILm0EEEEvSt12_Index_tupleIIXspT_EEE+0x43)[0x7fced3304077]
[node004:04351] [13] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EEclEv+0x2c)[0x7fced3303f7e]
[node004:04351] [14] [node004:04355] *** Process received signal ***
[node004:04355] Signal: Segmentation fault (11)
[node004:04355] Signal code: Invalid permissions (2)
[node004:04355] Failing at address: 0x7f5b22ffe410
[node002:04280] *** Process received signal ***
[node002:04280] Signal: Segmentation fault (11)
[node002:04280] Signal code: Invalid permissions (2)
[node002:04280] Failing at address: 0x7f85b9bf8000
[node002:04280] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f85f955a390]
[node002:04280] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x9f84e)[0x7f85f921e84e]
[node002:04280] [ 2] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso7UpdaterIfE6AccessEmPfS2_mPNS_9AddOptionE+0x4e)[0x7f85fa35f050]
[node002:04280] [ 3] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZN10multiverso17MatrixServerTableIfE10ProcessGetERKSt6vectorINS_4BlobESaIS3_EEPS5+0x39e)[0x7f85fa3491a0]
[node002:04280] [ 4] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso6Server10ProcessGetERSt10unique_ptrINS_7MessageESt14default_deleteIS2_EE+0x1df)[0x7f85fa33232d]
[node002:04280] [ 5] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso6ServerEFvRSt10unique_ptrINS0_7MessageESt14default_deleteIS3_EEELb1EEclIIS7_EvEEvPS1_DpOT+0x7c)[0x7f85fa3371c2]
[node002:04280] [ 6] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEE6__callIvJS8_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE+0x84)[0x7f85fa336bb4]
[node002:04280] [ 7] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEEclIJS8_EvEET0_DpOT+0x56)[0x7f85fa335dce]
[node002:04280] [ 8] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt17_Function_handlerIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEESt5_BindIFSt7_Mem_fnIMNS1_6ServerEFvS6_EEPSA_St12_PlaceholderILi1EEEEE9_M_invokeERKSt9_Any_dataS6+0x37)[0x7f85fa334f55]
[node002:04280] [ 9] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt8functionIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEEEclES6+0x49)[0x7f85fa314d8f]
[node002:04280] [10] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso5Actor4MainEv+0xf3)[0x7f85fa313153]
[node002:04280] [11] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso5ActorEFvvELb1EEclIJEvEEvPS1_DpOT+0x65)[0x7f85fa31a0e3]
[node002:04280] [12] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EE9_M_invokeIILm0EEEEvSt12_Index_tupleIIXspT_EEE+0x43)[0x7f85fa31a077]
[node002:04280] [13] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EEclEv+0x2c)[0x7f85fa319f7e]
[node002:04280] [14] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS4_EEE6_M_runEv+0x1c)[0x7f85fa319f0e]
[node002:04280] [15] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7f85f9f5fc80]
[node002:04280] [16] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f85f95506ba]
[node002:04280] [17] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f85f92863dd]
[node002:04280] *** End of error message ***
/home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS4_EEE6_M_runEv+0x1c)[0x7fced3303f0e]
[node004:04351] [15] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7fced2f49c80]
[node004:04351] [16] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fced253a6ba]
[node004:04351] [17] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fced22703dd]
[node004:04351] *** End of error message ***
[node004:04355] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f5b66ce1390]
[node004:04355] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x9f840)[0x7f5b669a5840]
[node004:04355] [ 2] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso7UpdaterIfE6AccessEmPfS2_mPNS_9AddOptionE+0x4e)[0x7f5b67ae6050]
[node004:04355] [ 3] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZN10multiverso17MatrixServerTableIfE10ProcessGetERKSt6vectorINS_4BlobESaIS3_EEPS5+0x39e)[0x7f5b67ad01a0]
[node004:04355] [ 4] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso6Server10ProcessGetERSt10unique_ptrINS_7MessageESt14default_deleteIS2_EE+0x1df)[0x7f5b67ab932d]
[node004:04355] [ 5] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso6ServerEFvRSt10unique_ptrINS0_7MessageESt14default_deleteIS3_EEELb1EEclIIS7_EvEEvPS1_DpOT+0x7c)[0x7f5b67abe1c2]
[node004:04355] [ 6] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEE6__callIvJS8_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE+0x84)[0x7f5b67abdbb4]
[node004:04355] [ 7] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEEclIJS8_EvEET0_DpOT+0x56)[0x7f5b67abcdce]
[node004:04355] [ 8] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt17_Function_handlerIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEESt5_BindIFSt7_Mem_fnIMNS1_6ServerEFvS6_EEPSA_St12_PlaceholderILi1EEEEE9_M_invokeERKSt9_Any_dataS6+0x37)[0x7f5b67abbf55]
[node004:04355] [ 9] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt8functionIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEEEclES6+0x49)[0x7f5b67a9bd8f]
[node004:04355] [10] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso5Actor4MainEv+0xf3)[0x7f5b67a9a153]
[node004:04355] [11] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso5ActorEFvvELb1EEclIJEvEEvPS1_DpOT+0x65)[0x7f5b67aa10e3]
[node004:04355] [12] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EE9_M_invokeIILm0EEEEvSt12_Index_tupleIIXspT_EEE+0x43)[0x7f5b67aa1077]
[node004:04355] [13] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EEclEv+0x2c)[0x7f5b67aa0f7e]
[node004:04355] [14] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS4_EEE6_M_runEv+0x1c)[0x7f5b67aa0f0e]
[node004:04355] [15] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7f5b676e6c80]
[node004:04355] [16] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f5b66cd76ba]
[node004:04355] [17] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5b66a0d3dd]
[node004:04355] *** End of error message ***
[node004:04346] *** Process received signal ***
[node004:04346] Signal: Segmentation fault (11)
[node004:04346] Signal code: Invalid permissions (2)
[node004:04346] Failing at address: 0x7f1ea6168000
[node004:04346] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f1ee59e9390]
[node004:04346] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x9f84e)[0x7f1ee56ad84e]
[node004:04346] [ 2] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso7UpdaterIfE6AccessEmPfS2_mPNS_9AddOptionE+0x4e)[0x7f1ee67ee050]
[node004:04346] [ 3] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZN10multiverso17MatrixServerTableIfE10ProcessGetERKSt6vectorINS_4BlobESaIS3_EEPS5+0x39e)[0x7f1ee67d81a0]
[node004:04346] [ 4] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso6Server10ProcessGetERSt10unique_ptrINS_7MessageESt14default_deleteIS2_EE+0x1df)[0x7f1ee67c132d]
[node004:04346] [ 5] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso6ServerEFvRSt10unique_ptrINS0_7MessageESt14default_deleteIS3_EEELb1EEclIIS7_EvEEvPS1_DpOT+0x7c)[0x7f1ee67c61c2]
[node004:04346] [ 6] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEE6__callIvJS8_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE+0x84)[0x7f1ee67c5bb4]
[node004:04346] [ 7] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEEclIJS8_EvEET0_DpOT+0x56)[0x7f1ee67c4dce]
[node004:04346] [ 8] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt17_Function_handlerIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEESt5_BindIFSt7_Mem_fnIMNS1_6ServerEFvS6_EEPSA_St12_PlaceholderILi1EEEEE9_M_invokeERKSt9_Any_dataS6+0x37)[0x7f1ee67c3f55]
[node004:04346] [ 9] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt8functionIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEEEclES6+0x49)[0x7f1ee67a3d8f]
[node004:04346] [10] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso5Actor4MainEv+0xf3)[0x7f1ee67a2153]
[node004:04346] [11] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso5ActorEFvvELb1EEclIJEvEEvPS1_DpOT+0x65)[0x7f1ee67a90e3]
[node004:04346] [12] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EE9_M_invokeIILm0EEEEvSt12_Index_tupleIIXspT_EEE+0x43)[0x7f1ee67a9077]
[node004:04346] [13] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EEclEv+0x2c)[0x7f1ee67a8f7e]
[node004:04346] [14] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS4_EEE6_M_runEv+0x1c)[0x7f1ee67a8f0e]
[node004:04346] [15] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7f1ee63eec80]
[node004:04346] [16] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f1ee59df6ba]
[node004:04346] [17] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f1ee57153dd]
[node004:04346] *** End of error message ***
[node002:04281] *** Process received signal ***
[node002:04281] Signal: Segmentation fault (11)
[node002:04281] Signal code: Invalid permissions (2)
[node002:04281] Failing at address: 0x7f14cdbf8410
[node002:04281] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f150d4da390]
[node002:04281] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x9f840)[0x7f150d19e840]
[node002:04281] [ 2] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso7UpdaterIfE6AccessEmPfS2_mPNS_9AddOptionE+0x4e)[0x7f150e2df050]
[node002:04281] [ 3] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZN10multiverso17MatrixServerTableIfE10ProcessGetERKSt6vectorINS_4BlobESaIS3_EEPS5+0x39e)[0x7f150e2c91a0]
[node002:04281] [ 4] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso6Server10ProcessGetERSt10unique_ptrINS_7MessageESt14default_deleteIS2_EE+0x1df)[0x7f150e2b232d]
[node002:04281] [ 5] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso6ServerEFvRSt10unique_ptrINS0_7MessageESt14default_deleteIS3_EEELb1EEclIIS7_EvEEvPS1_DpOT+0x7c)[0x7f150e2b71c2]
[node002:04281] [ 6] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEE6__callIvJS8_EJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE+0x84)[0x7f150e2b6bb4]
[node002:04281] [ 7] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt5_BindIFSt7_Mem_fnIMN10multiverso6ServerEFvRSt10unique_ptrINS1_7MessageESt14default_deleteIS4_EEEEPS2_St12_PlaceholderILi1EEEEclIJS8_EvEET0_DpOT+0x56)[0x7f150e2b5dce]
[node002:04281] [ 8] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNSt17_Function_handlerIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEESt5_BindIFSt7_Mem_fnIMNS1_6ServerEFvS6_EEPSA_St12_PlaceholderILi1EEEEE9_M_invokeERKSt9_Any_dataS6+0x37)[0x7f150e2b4f55]
[node002:04281] [ 9] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt8functionIFvRSt10unique_ptrIN10multiverso7MessageESt14default_deleteIS2_EEEEclES6+0x49)[0x7f150e294d8f]
[node002:04281] [10] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZN10multiverso5Actor4MainEv+0xf3)[0x7f150e293153]
[node002:04281] [11] /home/ubuntu/multiverso/build/src/libmultiverso.so(ZNKSt12_Mem_fn_baseIMN10multiverso5ActorEFvvELb1EEclIJEvEEvPS1_DpOT+0x65)[0x7f150e29a0e3]
[node002:04281] [12] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EE9_M_invokeIILm0EEEEvSt12_Index_tupleIIXspT_EEE+0x43)[0x7f150e29a077]
[node002:04281] [13] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS2_EEclEv+0x2c)[0x7f150e299f7e]
[node002:04281] [14] /home/ubuntu/multiverso/build/src/libmultiverso.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt7_Mem_fnIMN10multiverso5ActorEFvvEEPS4_EEE6_M_runEv+0x1c)[0x7f150e299f0e]
[node002:04281] [15] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7f150dedfc80]
[node002:04281] [16] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f150d4d06ba]
[node002:04281] [17] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f150d2063dd]
[node002:04281] *** End of error message ***
[INFO] [2017-10-10 07:43:14] Rank 20 Prepare data time:44.863608s
[node004][[21269,1],55][btl_tcp_frag.c:237:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)
[node004:04348] [[21269,1],55] ORTE_ERROR_LOG: Unreachable in file pml_ob1_sendreq.c at line 1130
[INFO] [2017-10-10 07:43:14] Rank 26 Prepare data time:44.876433s

mpirun noticed that process rank 22 with PID 0 on node node002 exited on signal 11 (Segmentation fault).

[WalkerWen]

I met the same problem. How do you solve it? Thanks