libMesh::LibMeshInit Class Reference

#include <libmesh.h>

Public Member Functions

 LibMeshInit (int argc, const char *const *argv, MPI_Comm COMM_WORLD_IN=MPI_COMM_WORLD)
 
 LibMeshInit (int argc, const char *const *argv)
 
virtual ~LibMeshInit ()
 
const Parallel::Communicatorcomm () const
 
Parallel::Communicatorcomm ()
 

Private Attributes

Parallel::Communicator _comm
 
vtkMPIController * _vtk_mpi_controller
 

Detailed Description

The LibMeshInit class, when constructed, initializes the dependent libraries (e.g. MPI or PETSC) and does the command line parsing needed by libMesh. The LibMeshInit destructor closes those libraries properly.

For most users, a single LibMeshInit object should be created at the start of your main() function. This object replaces the previous

libMesh::init()/libMesh::close()

methods, which are now deprecated.

Definition at line 62 of file libmesh.h.

Constructor & Destructor Documentation

libMesh::LibMeshInit::LibMeshInit ( int  argc,
const char *const *  argv,
MPI_Comm  COMM_WORLD_IN = MPI_COMM_WORLD 
)

Initialize the library for use, with the command line options provided. This will e.g. call PetscInitialize if PETSC is available. You must create a LibMeshInit object before using any of the library functionality. This method may take an optional parameter to use a user-specified MPI communicator.

libMesh::LibMeshInit::LibMeshInit ( int  argc,
const char *const *  argv 
)

Definition at line 343 of file libmesh.C.

References libMesh::libMeshPrivateData::_is_initialized, libMesh::libMeshPrivateData::_n_processors, libMesh::libMeshPrivateData::_processor_id, libMesh::Singleton::cleanup(), libMesh::PerfLog::clear(), libMesh::Parallel::Communicator::clear(), libMesh::closed(), libMesh::COMM_WORLD, libMesh::command_line_value(), libMesh::PerfLog::disable_logging(), libMesh::ReferenceCounter::disable_print_counter_info(), DMCreate_libMesh(), libMesh::enableFPE(), libMesh::enableSEGV(), libMesh::err, libMesh::BasicOStreamProxy< charT, traits >::get(), libMesh::GLOBAL_COMM_WORLD, libMesh::global_processor_id(), ierr, libMesh::initialized(), libMesh::libmesh_assert(), libMesh_MPI_Handler(), libmesh_nullptr, libMesh::libmesh_terminate_handler(), libMesh::ReferenceCounter::n_objects(), libMesh::n_threads(), libMesh::on_command_line(), libMesh::out, libMesh::perflog, libMesh::ReferenceCounter::print_info(), libMesh::PerfLog::print_log(), libMesh::BasicOStreamProxy< charT, traits >::rdbuf(), libMesh::remote_elem, libMesh::BasicOStreamProxy< charT, traits >::reset(), libMesh::Singleton::setup(), and ~LibMeshInit().

348 {
349  // should _not_ be initialized already.
351 
352  // Build a command-line parser.
353  command_line.reset (new GetPot (argc, argv));
354 
355  // Disable performance logging upon request
356  {
357  if (libMesh::on_command_line ("--disable-perflog"))
359  }
360 
361  // Build a task scheduler
362  {
363  // Get the requested number of threads, defaults to 1 to avoid MPI and
364  // multithreading competition. If you would like to use MPI and multithreading
365  // at the same time then (n_mpi_processes_per_node)x(n_threads) should be the
366  // number of processing cores per node.
367  std::vector<std::string> n_threads(2);
368  n_threads[0] = "--n_threads";
369  n_threads[1] = "--n-threads";
372 
373  // If there's no threading model active, force _n_threads==1
374 #if !LIBMESH_USING_THREADS
376  {
378  libmesh_warning("Warning: You requested --n-threads>1 but no threading model is active!\n"
379  << "Forcing --n-threads==1 instead!");
380  }
381 #endif
382 
383  // Set the number of OpenMP threads to the same as the number of threads libMesh is going to use
384 #ifdef LIBMESH_HAVE_OPENMP
385  omp_set_num_threads(libMesh::libMeshPrivateData::_n_threads);
386 #endif
387 
388  task_scheduler.reset (new Threads::task_scheduler_init(libMesh::n_threads()));
389  }
390 
391  // Construct singletons who may be at risk of the
392  // "static initialization order fiasco"
394 
395  // Make sure the construction worked
397 
398 #if defined(LIBMESH_HAVE_MPI)
399 
400  // Allow the user to bypass MPI initialization
401  if (!libMesh::on_command_line ("--disable-mpi"))
402  {
403  // Check whether the calling program has already initialized
404  // MPI, and avoid duplicate Init/Finalize
405  int flag;
406  libmesh_call_mpi(MPI_Initialized (&flag));
407 
408  if (!flag)
409  {
410 #if MPI_VERSION > 1
411  int mpi_thread_provided;
412  const int mpi_thread_requested = libMesh::n_threads() > 1 ?
413  MPI_THREAD_FUNNELED :
414  MPI_THREAD_SINGLE;
415 
416  libmesh_call_mpi
417  (MPI_Init_thread (&argc, const_cast<char ***>(&argv),
418  mpi_thread_requested, &mpi_thread_provided));
419 
420  if ((libMesh::n_threads() > 1) &&
421  (mpi_thread_provided < MPI_THREAD_FUNNELED))
422  {
423  libmesh_warning("Warning: MPI failed to guarantee MPI_THREAD_FUNNELED\n"
424  << "for a threaded run.\n"
425  << "Be sure your library is funneled-thread-safe..."
426  << std::endl);
427 
428  // Ideally, if an MPI stack tells us it's unsafe for us
429  // to use threads, we shouldn't use threads.
430  // In practice, we've encountered one MPI stack (an
431  // mvapich2 configuration) that returned
432  // MPI_THREAD_SINGLE as a proper warning, two stacks
433  // that handle MPI_THREAD_FUNNELED properly, and two
434  // current stacks plus a couple old stacks that return
435  // MPI_THREAD_SINGLE but support libMesh threaded runs
436  // anyway.
437 
438  // libMesh::libMeshPrivateData::_n_threads = 1;
439  // task_scheduler.reset (new Threads::task_scheduler_init(libMesh::n_threads()));
440  }
441 #else
443  {
444  libmesh_warning("Warning: using MPI1 for threaded code.\n" <<
445  "Be sure your library is funneled-thread-safe..." <<
446  std::endl);
447  }
448 
449  libmesh_call_mpi
450  (MPI_Init (&argc, const_cast<char ***>(&argv)));
451 #endif
452  libmesh_initialized_mpi = true;
453  }
454 
455  // Duplicate the input communicator for internal use
456  // And get a Parallel::Communicator copy too, to use
457  // as a default for that API
458  this->_comm = COMM_WORLD_IN;
459 
460  libMesh::GLOBAL_COMM_WORLD = COMM_WORLD_IN;
461 
462 #ifndef LIBMESH_DISABLE_COMMWORLD
463  libMesh::COMM_WORLD = COMM_WORLD_IN;
464  Parallel::Communicator_World = COMM_WORLD_IN;
465 #endif
466 
467  //MPI_Comm_set_name not supported in at least SGI MPT's MPI implementation
468  //MPI_Comm_set_name (libMesh::COMM_WORLD, "libMesh::COMM_WORLD");
469 
471  cast_int<processor_id_type>(this->comm().rank());
473  cast_int<processor_id_type>(this->comm().size());
474 
475  // Set up an MPI error handler if requested. This helps us get
476  // into a debugger with a proper stack when an MPI error occurs.
477  if (libMesh::on_command_line ("--handle-mpi-errors"))
478  {
479 #if MPI_VERSION > 1
480  libmesh_call_mpi
481  (MPI_Comm_create_errhandler(libMesh_MPI_Handler, &libmesh_errhandler));
482  libmesh_call_mpi
483  (MPI_Comm_set_errhandler(libMesh::GLOBAL_COMM_WORLD, libmesh_errhandler));
484  libmesh_call_mpi
485  (MPI_Comm_set_errhandler(MPI_COMM_WORLD, libmesh_errhandler));
486 #else
487  libmesh_call_mpi
488  (MPI_Errhandler_create(libMesh_MPI_Handler, &libmesh_errhandler));
489  libmesh_call_mpi
490  (MPI_Errhandler_set(libMesh::GLOBAL_COMM_WORLD, libmesh_errhandler));
491  libmesh_call_mpi
492  (MPI_Errhandler_set(MPI_COMM_WORLD, libmesh_errhandler));
493 #endif // #if MPI_VERSION > 1
494  }
495  }
496 
497  // Could we have gotten bad values from the above calls?
498  libmesh_assert_greater (libMeshPrivateData::_n_processors, 0);
499 
500  // The cast_int already tested _processor_id>=0
501  // libmesh_assert_greater_equal (libMeshPrivateData::_processor_id, 0);
502 
503  // Let's be sure we properly initialize on every processor at once:
504  libmesh_parallel_only(this->comm());
505 
506 #endif
507 
508 #if defined(LIBMESH_HAVE_PETSC)
509 
510  // Allow the user to bypass PETSc initialization
511  if (!libMesh::on_command_line ("--disable-petsc")
512 
513 #if defined(LIBMESH_HAVE_MPI)
514  // If the user bypassed MPI, we'd better be safe and assume that
515  // PETSc was built to require it; otherwise PETSc initialization
516  // dies.
517  && !libMesh::on_command_line ("--disable-mpi")
518 #endif
519  )
520  {
521  int ierr=0;
522 
523  PETSC_COMM_WORLD = libMesh::GLOBAL_COMM_WORLD;
524 
525  // Check whether the calling program has already initialized
526  // PETSc, and avoid duplicate Initialize/Finalize
527  PetscBool petsc_already_initialized;
528  ierr = PetscInitialized(&petsc_already_initialized);
529  CHKERRABORT(libMesh::GLOBAL_COMM_WORLD,ierr);
530  if (petsc_already_initialized != PETSC_TRUE)
531  libmesh_initialized_petsc = true;
532 # if defined(LIBMESH_HAVE_SLEPC)
533 
534  // If SLEPc allows us to check whether the calling program
535  // has already initialized it, we do that, and avoid
536  // duplicate Initialize/Finalize.
537  // We assume that SLEPc will handle PETSc appropriately,
538  // which it does in the versions we've checked.
539  if (!SlepcInitializeCalled)
540  {
541  ierr = SlepcInitialize (&argc, const_cast<char ***>(&argv), libmesh_nullptr, libmesh_nullptr);
542  CHKERRABORT(libMesh::GLOBAL_COMM_WORLD,ierr);
543  libmesh_initialized_slepc = true;
544  }
545 # else
546  if (libmesh_initialized_petsc)
547  {
548  ierr = PetscInitialize (&argc, const_cast<char ***>(&argv), libmesh_nullptr, libmesh_nullptr);
549  CHKERRABORT(libMesh::GLOBAL_COMM_WORLD,ierr);
550  }
551 # endif
552 #if !PETSC_RELEASE_LESS_THAN(3,3,0)
553  // Register the reference implementation of DMlibMesh
554 #if PETSC_RELEASE_LESS_THAN(3,4,0)
555  ierr = DMRegister(DMLIBMESH, PETSC_NULL, "DMCreate_libMesh", DMCreate_libMesh); CHKERRABORT(libMesh::GLOBAL_COMM_WORLD,ierr);
556 #else
557  ierr = DMRegister(DMLIBMESH, DMCreate_libMesh); CHKERRABORT(libMesh::GLOBAL_COMM_WORLD,ierr);
558 #endif
559 
560 #endif
561  }
562 #endif
563 
564 #if defined(LIBMESH_HAVE_MPI) && defined(LIBMESH_HAVE_VTK)
565  // Do MPI initializtion for VTK.
566  _vtk_mpi_controller = vtkMPIController::New();
567  _vtk_mpi_controller->Initialize(&argc, const_cast<char ***>(&argv), /*initialized_externally=*/1);
568  _vtk_mpi_controller->SetGlobalController(_vtk_mpi_controller);
569 #endif
570 
571  // Re-parse the command-line arguments. Note that PETSc and MPI
572  // initialization above may have removed command line arguments
573  // that are not relevant to this application in the above calls.
574  // We don't want a false-positive by detecting those arguments.
575  //
576  // Note: this seems overly paranoid/like it should be unnecessary,
577  // plus we were doing it wrong for many years and not clearing the
578  // existing GetPot object before re-parsing the command line, so all
579  // the command line arguments appeared twice in the GetPot object...
580  command_line.reset (new GetPot (argc, argv));
581 
582  // The following line is an optimization when simultaneous
583  // C and C++ style access to output streams is not required.
584  // The amount of benefit which occurs is probably implementation
585  // defined, and may be nothing. On the other hand, I have seen
586  // some IO tests where IO peformance improves by a factor of two.
587  if (!libMesh::on_command_line ("--sync-with-stdio"))
588  std::ios::sync_with_stdio(false);
589 
590  // Honor the --separate-libmeshout command-line option.
591  // When this is specified, the library uses an independent ostream
592  // for libMesh::out/libMesh::err messages, and
593  // std::cout and std::cerr are untouched by any other options
594  if (libMesh::on_command_line ("--separate-libmeshout"))
595  {
596  // Redirect. We'll share streambufs with cout/cerr for now, but
597  // presumably anyone using this option will want to replace the
598  // bufs later.
599  std::ostream * newout = new std::ostream(std::cout.rdbuf());
600  libMesh::out = *newout;
601  std::ostream * newerr = new std::ostream(std::cerr.rdbuf());
602  libMesh::err = *newerr;
603  }
604 
605  // Process command line arguments for redirecting stdout/stderr.
606  bool
607  cmdline_has_redirect_stdout = libMesh::on_command_line ("--redirect-stdout"),
608  cmdline_has_redirect_output = libMesh::on_command_line ("--redirect-output");
609 
610  // The --redirect-stdout command-line option has been deprecated in
611  // favor of "--redirect-output basename".
612  if (cmdline_has_redirect_stdout)
613  libmesh_warning("The --redirect-stdout command line option has been deprecated. "
614  "Use '--redirect-output basename' instead.");
615 
616  // Honor the "--redirect-stdout" and "--redirect-output basename"
617  // command-line options. When one of these is specified, each
618  // processor sends libMesh::out/libMesh::err messages to
619  // stdout.processor.#### (default) or basename.processor.####.
620  if (cmdline_has_redirect_stdout || cmdline_has_redirect_output)
621  {
622  std::string basename = "stdout";
623 
624  // Look for following argument if using new API
625  if (cmdline_has_redirect_output)
626  {
627  // Set the cursor to the correct location in the list of command line arguments.
628  command_line->search(1, "--redirect-output");
629 
630  // Get the next option on the command line as a string.
631  std::string next_string = "";
632  next_string = command_line->next(next_string);
633 
634  // If the next string starts with a dash, we assume it's
635  // another flag and not a file basename requested by the
636  // user.
637  if (next_string.size() > 0 && next_string.find_first_of("-") != 0)
638  basename = next_string;
639  }
640 
641  std::ostringstream filename;
642  filename << basename << ".processor." << libMesh::global_processor_id();
643  _ofstream.reset (new std::ofstream (filename.str().c_str()));
644 
645  // Redirect, saving the original streambufs!
646  out_buf = libMesh::out.rdbuf (_ofstream->rdbuf());
647  err_buf = libMesh::err.rdbuf (_ofstream->rdbuf());
648  }
649 
650  // redirect libMesh::out to nothing on all
651  // other processors unless explicitly told
652  // not to via the --keep-cout command-line argument.
653  if (libMesh::global_processor_id() != 0)
654  if (!libMesh::on_command_line ("--keep-cout"))
656 
657  // Similarly, the user can request to drop cerr on all non-0 ranks.
658  // By default, errors are printed on all ranks, but this can lead to
659  // interleaved/unpredictable outputs when doing parallel regression
660  // testing, which this option is designed to support.
661  if (libMesh::global_processor_id() != 0)
662  if (libMesh::on_command_line ("--drop-cerr"))
664 
665  // Check command line to override printing
666  // of reference count information.
667  if(libMesh::on_command_line("--disable-refcount-printing") )
669 
670 #ifdef LIBMESH_ENABLE_EXCEPTIONS
671  // Set our terminate handler to write stack traces in the event of a
672  // crash
673  old_terminate_handler = std::set_terminate(libmesh_terminate_handler);
674 #endif
675 
676 
677  if (libMesh::on_command_line("--enable-fpe"))
678  libMesh::enableFPE(true);
679 
680  if (libMesh::on_command_line("--enable-segv"))
681  libMesh::enableSEGV(true);
682 
683  // The library is now ready for use
685 
686 
687  // Make sure these work. Library methods
688  // depend on these being implemented properly,
689  // so this is a good time to test them!
692 }
693 
694 
695 
697 {
698  // Every processor had better be ready to exit at the same time.
699  // This would be a libmesh_parallel_only() function, except that
700  // libmesh_parallel_only() uses libmesh_assert() which throws an
701  // exception() which causes compilers to scream about exceptions
702  // inside destructors.
703 
704  // Even if we're not doing parallel_only debugging, we don't want
705  // one processor to try to exit until all others are done working.
706  this->comm().barrier();
707 
708  // We can't delete, finalize, etc. more than once without
709  // reinitializing in between
710  libmesh_exceptionless_assert(!libMesh::closed());
711 
712  // Delete reference counted singleton(s)
714 
715  // Clear the thread task manager we started
716  task_scheduler.reset();
717 
718  // Force the \p ReferenceCounter to print
719  // its reference count information. This allows
720  // us to find memory leaks. By default the
721  // \p ReferenceCounter only prints its information
722  // when the last created object has been destroyed.
723  // That does no good if we are leaking memory!
725 
726 
727  // Print an informative message if we detect a memory leak
728  if (ReferenceCounter::n_objects() != 0)
729  {
730  libMesh::err << "Memory leak detected!"
731  << std::endl;
732 
733 #if !defined(LIBMESH_ENABLE_REFERENCE_COUNTING) || defined(NDEBUG)
734 
735  libMesh::err << "Compile in DEBUG mode with --enable-reference-counting"
736  << std::endl
737  << "for more information"
738  << std::endl;
739 #endif
740 
741  }
742 
743  // print the perflog to individual processor's file.
745 
746  // Now clear the logging object, we don't want it to print
747  // a second time during the PerfLog destructor.
749 
750  // Reconnect the output streams
751  // (don't do this, or we will get messages from objects
752  // that go out of scope after the following return)
753  //std::cout.rdbuf(std::cerr.rdbuf());
754 
755 
756  // Set the initialized() flag to false
758 
759  if (libMesh::on_command_line ("--redirect-stdout") ||
760  libMesh::on_command_line ("--redirect-output"))
761  {
762  // If stdout/stderr were redirected to files, reset them now.
763  libMesh::out.rdbuf (out_buf);
764  libMesh::err.rdbuf (err_buf);
765  }
766 
767  // If we built our own output streams, we want to clean them up.
768  if (libMesh::on_command_line ("--separate-libmeshout"))
769  {
770  delete libMesh::out.get();
771  delete libMesh::err.get();
772 
773  libMesh::out.reset(std::cout);
774  libMesh::err.reset(std::cerr);
775  }
776 
777 #ifdef LIBMESH_ENABLE_EXCEPTIONS
778  // Reset the old terminate handler; maybe the user code wants to
779  // keep doing C++ stuff after closing libMesh stuff.
780  std::set_terminate(old_terminate_handler);
781 #endif
782 
783 
784  if (libMesh::on_command_line("--enable-fpe"))
785  libMesh::enableFPE(false);
786 
787 #if defined(LIBMESH_HAVE_PETSC)
788  // Allow the user to bypass PETSc finalization
789  if (!libMesh::on_command_line ("--disable-petsc")
790 #if defined(LIBMESH_HAVE_MPI)
791  && !libMesh::on_command_line ("--disable-mpi")
792 #endif
793  )
794  {
795 # if defined(LIBMESH_HAVE_SLEPC)
796  if (libmesh_initialized_slepc)
797  SlepcFinalize();
798 # else
799  if (libmesh_initialized_petsc)
800  PetscFinalize();
801 # endif
802  }
803 #endif
804 
805 #if defined(LIBMESH_HAVE_MPI) && defined(LIBMESH_HAVE_VTK)
806  _vtk_mpi_controller->Finalize(/*finalized_externally=*/1);
807  _vtk_mpi_controller->Delete();
808 #endif
809 
810 #if defined(LIBMESH_HAVE_MPI)
811  // Allow the user to bypass MPI finalization
812  if (!libMesh::on_command_line ("--disable-mpi"))
813  {
814  this->_comm.clear();
815 #ifndef LIBMESH_DISABLE_COMMWORLD
817 #endif
818 
819  if (libmesh_initialized_mpi)
820  {
821  // We can't just libmesh_assert here because destructor,
822  // but we ought to report any errors
823  unsigned int error_code = MPI_Finalize();
824  if (error_code != MPI_SUCCESS)
825  {
826  char error_string[MPI_MAX_ERROR_STRING+1];
827  int error_string_len;
828  MPI_Error_string(error_code, error_string,
829  &error_string_len);
830  std::cerr << "Failure from MPI_Finalize():\n"
831  << error_string << std::endl;
832  }
833  }
834  }
835 #endif
836 }
bool closed()
Definition: libmesh.C:279
static unsigned int n_objects()
unsigned int n_threads()
Definition: libmesh_base.h:125
FakeCommunicator & Communicator_World
unsigned int size() const
Definition: parallel.h:679
streambufT * rdbuf() const
void enableSEGV(bool on)
Definition: libmesh.C:888
processor_id_type _n_processors
Definition: libmesh.C:249
Parallel::Communicator _comm
Definition: libmesh.h:86
const class libmesh_nullptr_t libmesh_nullptr
MPI_Comm GLOBAL_COMM_WORLD
Definition: libmesh.C:197
MPI_Comm COMM_WORLD
Definition: libmesh.C:195
void reset(streamT &target)
vtkMPIController * _vtk_mpi_controller
Definition: libmesh.h:92
libmesh_assert(j)
PetscErrorCode DMCreate_libMesh(DM dm)
tbb::task_scheduler_init task_scheduler_init
Definition: threads_tbb.h:73
std::terminate_handler old_terminate_handler
Definition: libmesh.C:286
processor_id_type _processor_id
Definition: libmesh.C:250
const Parallel::Communicator & comm() const
Definition: libmesh.h:81
OStreamProxy err(std::cerr)
static void print_info(std::ostream &out=libMesh::out)
PerfLog perflog("libMesh",#ifdef LIBMESH_ENABLE_PERFORMANCE_LOGGING true#else false#endif)
MPI_Errhandler libmesh_errhandler
Definition: libmesh.C:247
void clear()
Definition: perf_log.C:67
void enableFPE(bool on)
Definition: libmesh.C:843
PetscTruth PetscBool
Definition: petsc_macro.h:64
PetscErrorCode ierr
bool on_command_line(const std::string &arg)
Definition: libmesh.C:914
processor_id_type global_processor_id()
Definition: libmesh_base.h:114
bool initialized()
Definition: libmesh.C:272
void libMesh_MPI_Handler(MPI_Comm *, int *,...)
Definition: libmesh.C:160
unsigned int rank() const
Definition: parallel.h:677
static void cleanup()
void print_log() const
Definition: perf_log.C:582
T command_line_value(const std::string &name, T value)
Definition: libmesh.C:925
static void disable_print_counter_info()
OStreamProxy out(std::cout)
void disable_logging()
Definition: perf_log.h:155
void libmesh_terminate_handler()
Definition: libmesh.C:288
const RemoteElem * remote_elem
Definition: remote_elem.C:57
virtual libMesh::LibMeshInit::~LibMeshInit ( )
virtual

Referenced by LibMeshInit().

Member Function Documentation

const Parallel::Communicator& libMesh::LibMeshInit::comm ( ) const
inline

Definition at line 81 of file libmesh.h.

References _comm.

81 { return _comm; }
Parallel::Communicator _comm
Definition: libmesh.h:86
Parallel::Communicator& libMesh::LibMeshInit::comm ( )
inline

Definition at line 83 of file libmesh.h.

References _comm.

83 { return _comm; }
Parallel::Communicator _comm
Definition: libmesh.h:86

Member Data Documentation

Parallel::Communicator libMesh::LibMeshInit::_comm
private

Definition at line 86 of file libmesh.h.

Referenced by comm().

vtkMPIController* libMesh::LibMeshInit::_vtk_mpi_controller
private

Definition at line 92 of file libmesh.h.


The documentation for this class was generated from the following files: