walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T > Class Template Reference

Detailed Description

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
class walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >

Manages MPI Communication with a set of known communication partners.

Features / Restrictions:

  • usable in every case where normal MPI_Send, MPI_Recv is needed
  • communication partners have to be known, message size not necessarily
  • unknown message sizes possible ( -> automatic extra message to exchange sizes )
  • Implemented with non-blocking MPI calls, and MPI_Waitany to process a message as soon as it was received while still waiting for other messages

Example:

BufferSystem bs ( MPI_COMM_WORLD );
// we assume every process has exactly two neighbors ( periodic )
// and the neighbor ranks are stored in 'neighborRank1' and 'neighborRank2'
bs.sendBuffer( neighborRank1 ) << 42;
bs.sendBuffer( neighborRank2 ) << 4242;
// We expect to receive the same amount as we send:
bs.setReceiverInfoFromSendBufferState( true, false );
bs.sendAll();
for ( auto i = bs.begin(); i != bs.end(); ++i ) {
cout << "Message received from " << i.rank() << endl;
int messageContent;
i.buffer() >> messageContent;
cout << "Received " << messageContent << endl;
}

Usage:

  1. Setup:
    • define receive information using setReceiverInfo() or setReceiverInfoFromSendBufferState() With these functions one defines the processes that we receive from and optionally the size of the received messages. If the message sizes are unknown, they have to be communicated first. One also defines if the sizes stay constant or if they change in each communication step (size message is then sent before every content message)
    • The receiver and send information can be changed, if no communication is currently running.
  2. Communication Step:
    • Optionally call scheduleReceives() -> starts communication step and causes MPI_IRecv's to be called. This is also automatically called on first send operation.
    • fill send buffers
    • call send() for each buffer after filling the buffer, or call sendAll() after filling all buffers. The send*() functions return immediately, internally MPI_ISend's are called. The send*() functions start the communication step if it was not already started by scheduleReceives()
    • Receiving: iterate over incoming messages using begin() and end(). Internally a MPI_Waitany is executed that returns as soon as a single message was received. Then this message can be processed while waiting for the other messages.
      Attention
      Even if the current process does not receive anything, call begin() and end() to finish the communication step
    • When iteration has reached the end the communication step is finished.
    • when communication has finished all Send- and RecvBuffers are automatically cleared

When running multiple BufferSystems concurrently different MPI tags have to be used for the systems: the tag can be passed in the constructor.

#include <BufferSystem.h>

Classes

class  iterator
 
struct  SendInfo
 

Public Member Functions

void useIProbe (const bool use)
 
bool isIProbedUsed () const
 
int64_t getBytesSent () const
 Bytes sent during the current or last communication. More...
 
int64_t getBytesReceived () const
 Bytes received during the current or last communication. More...
 
int64_t getNumberOfSends () const
 Communication partners during current or last send operation. More...
 
int64_t getNumberOfReceives () const
 Communication partners during current or last receive operation. More...
 
Constructors
 GenericBufferSystem (const MPI_Comm &communicator, int tag=0)
 
 GenericBufferSystem (const GenericBufferSystem &other)
 
GenericBufferSystemoperator= (const GenericBufferSystem &other)
 
 ~GenericBufferSystem ()
 
Receiver Registration
template<typename RankIter >
void setReceiverInfo (RankIter begin, RankIter end, bool changingSize)
 
template<typename Range >
void setReceiverInfo (const Range &range, bool changingSize)
 
void setReceiverInfo (const std::set< MPIRank > &ranksToRecvFrom, bool changingSize)
 Sets receiver information, when message sizes are unknown. More...
 
void setReceiverInfo (const std::map< MPIRank, MPISize > &ranksToRecvFrom)
 Sets receiver information, when message sizes are known. More...
 
void setReceiverInfoFromSendBufferState (bool useSizeFromSendBuffers, bool changingSize)
 Sets receiver information, using SendBuffers (symmetric communication) More...
 
void sizeHasChanged (bool alwaysChangingSize=false)
 Notifies that GenericBufferSystem that message sizes have changed ( and optionally are changing in all following steps) More...
 
Executing Communication Step
void scheduleReceives ()
 
SendBuffer_T & sendBuffer (MPIRank rank)
 Returns an existing SendBuffer, or creates a new one (only if !isCommunicationRunning() ) More...
 
SendBuffer_T & sendBuffer (uint_t rank)
 
size_t size () const
 
void sendAll ()
 Sends filled ( nonzero length) SendBuffers to their destination ranks. More...
 
void send (MPIRank rank)
 Sends a single SendBuffer to its destination rank. More...
 
iterator begin ()
 
iterator end ()
 
Status Queries
bool isSizeCommunicatedInNextStep () const
 
bool isCommunicationRunning () const
 
bool isReceiverInformationSet () const
 

Protected Member Functions

void startCommunication ()
 Starts communication. More...
 
void endCommunication ()
 Cleanup after communication. More...
 
RecvBuffer_T * waitForNext (MPIRank &fromRank)
 Helper function for iterator. More...
 
void setCommunicationType (const bool knownSize)
 Sets the communication type to known size, unknown size or NoMPI comm. More...
 

Protected Attributes

internal::KnownSizeCommunication< RecvBuffer_T, SendBuffer_T > knownSizeComm_
 
internal::UnknownSizeCommunication< RecvBuffer_T, SendBuffer_T > unknownSizeComm_
 
internal::UnknownSizeCommunicationIProbe< RecvBuffer_T, SendBuffer_T > unknownSizeCommIProbe_
 
internal::NoMPICommunication< RecvBuffer_T, SendBuffer_T > noMPIComm_
 
internal::AbstractCommunication< RecvBuffer_T, SendBuffer_T > * currentComm_
 
bool sizeChangesEverytime_
 
bool communicationRunning_
 
std::map< MPIRank, typename internal::AbstractCommunication< RecvBuffer_T, SendBuffer_T >::ReceiveInfo > recvInfos_
 Info about the message to be received from a certain rank: information holds the buffer and, if known, the message size. More...
 
std::map< MPIRank, SendInfosendInfos_
 
bool useIProbe_ = false
 switch betwenn IProbe and two message communication for varying size communication More...
 
int64_t bytesSent_ = 0
 number of bytes sent during last communication More...
 
int64_t bytesReceived_ = 0
 number of bytes received during last communication More...
 
int64_t numberOfSends_ = 0
 number of communication partners during last send More...
 
int64_t numberOfReceives_ = 0
 number of communication partners during last receive More...
 

Static Protected Attributes

static std::set< int > activeTags_
 

Friends

class GenericOpenMPBufferSystem< RecvBuffer_T, SendBuffer_T >
 
Iterator
class iterator
 

Rank Ranges

typedef std::set< MPIRank > RankRange
 
static RankRange noRanks ()
 
static RankRange allRanks ()
 
static RankRange allRanksButRoot ()
 
static RankRange onlyRoot ()
 
static RankRange onlyRank (MPIRank includedRank)
 

Member Typedef Documentation

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
typedef std::set<MPIRank> walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::RankRange

Constructor & Destructor Documentation

template<typename Rb , typename Sb >
walberla::mpi::GenericBufferSystem< Rb, Sb >::GenericBufferSystem ( const MPI_Comm &  communicator,
int  tag = 0 
)
explicit
template<typename Rb , typename Sb >
walberla::mpi::GenericBufferSystem< Rb, Sb >::GenericBufferSystem ( const GenericBufferSystem< RecvBuffer_T, SendBuffer_T > &  other)
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::~GenericBufferSystem ( )
inline

Member Function Documentation

template<typename Rb , typename Sb >
GenericBufferSystem< Rb, Sb >::RankRange walberla::mpi::GenericBufferSystem< Rb, Sb >::allRanks ( )
static
template<typename Rb , typename Sb >
GenericBufferSystem< Rb, Sb >::RankRange walberla::mpi::GenericBufferSystem< Rb, Sb >::allRanksButRoot ( )
static
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
iterator walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::begin ( )
inline
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
iterator walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::end ( )
inline
template<typename Rb , typename Sb >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::endCommunication ( )
protected

Cleanup after communication.

  • wait for sends to complete
  • clear buffers
  • manage sizeChangesEverytime
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
int64_t walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::getBytesReceived ( ) const
inline

Bytes received during the current or last communication.

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
int64_t walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::getBytesSent ( ) const
inline

Bytes sent during the current or last communication.

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
int64_t walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::getNumberOfReceives ( ) const
inline

Communication partners during current or last receive operation.

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
int64_t walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::getNumberOfSends ( ) const
inline

Communication partners during current or last send operation.

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
bool walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::isCommunicationRunning ( ) const
inline
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
bool walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::isIProbedUsed ( ) const
inline
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
bool walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::isReceiverInformationSet ( ) const
inline
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
bool walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::isSizeCommunicatedInNextStep ( ) const
inline
template<typename Rb , typename Sb >
GenericBufferSystem< Rb, Sb >::RankRange walberla::mpi::GenericBufferSystem< Rb, Sb >::noRanks ( )
static
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
GenericBufferSystem< Rb, Sb >::RankRange walberla::mpi::GenericBufferSystem< Rb, Sb >::onlyRank ( MPIRank  includedRank)
static
template<typename Rb , typename Sb >
GenericBufferSystem< Rb, Sb >::RankRange walberla::mpi::GenericBufferSystem< Rb, Sb >::onlyRoot ( )
static
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
GenericBufferSystem< Rb, Sb > & walberla::mpi::GenericBufferSystem< Rb, Sb >::operator= ( const GenericBufferSystem< RecvBuffer_T, SendBuffer_T > &  other)
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
void walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::scheduleReceives ( )
inline
template<typename Rb , typename Sb >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::send ( MPIRank  rank)

Sends a single SendBuffer to its destination rank.

If SendBuffer is empty no message is sent.

If communication was not started before, it is started with this function.

template<typename Rb , typename Sb >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::sendAll ( )

Sends filled ( nonzero length) SendBuffers to their destination ranks.

If some of the SendBuffers have been sent manually before using send(int rank) they are skipped, only the remaining buffers are sent.

If communication was not started before, it is started with this function.

template<typename Rb , typename Sb >
Sb & walberla::mpi::GenericBufferSystem< Rb, Sb >::sendBuffer ( MPIRank  rank)

Returns an existing SendBuffer, or creates a new one (only if !isCommunicationRunning() )

Parameters
rankthe rank where the buffer should be sent to
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
SendBuffer_T& walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::sendBuffer ( uint_t  rank)
inline
template<typename Rb , typename Sb >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::setCommunicationType ( const bool  knownSize)
protected

Sets the communication type to known size, unknown size or NoMPI comm.

template<typename Rb , typename Sb >
template<typename RankIter >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::setReceiverInfo ( RankIter  begin,
RankIter  end,
bool  changingSize 
)
template<typename Rb , typename Sb >
template<typename Range >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::setReceiverInfo ( const Range &  range,
bool  changingSize 
)
template<typename Rb , typename Sb >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::setReceiverInfo ( const std::set< MPIRank > &  ranksToRecvFrom,
bool  changingSize 
)

Sets receiver information, when message sizes are unknown.

Parameters
ranksToRecvFromset of all ranks where messages are received
changingSizetrue if the message size is different in each communication step. If false the message size is exchanged once and is expected to be constant If true the message size is exchanged before each communication step. The behavior can be changed later one using setReceiverInfo() or sizeHasChanged().
template<typename Rb , typename Sb >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::setReceiverInfo ( const std::map< MPIRank, MPISize > &  ranksToRecvFrom)

Sets receiver information, when message sizes are known.

Parameters
ranksToRecvFromMap containing all ranks, where messages are received from, as keys and the message sizes as values. The message sizes are expected to be constant for all communication step until behavior is changed with setReceiverInfo*() or sizeHasChanged()
template<typename Rb , typename Sb >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::setReceiverInfoFromSendBufferState ( bool  useSizeFromSendBuffers,
bool  changingSize 
)

Sets receiver information, using SendBuffers (symmetric communication)

Gives the GenericBufferSystem the information that messages are received from the same processes that we send to (i.e. from all ranks where SendBuffers were already filled ) sendBuffer() has to be called before, and corresponding SendBuffers have to be filled.

Parameters
useSizeFromSendBuffersIf true, the sizes are expected to be known and equal to the size of the SendBuffers. SendBuffers with zero size are ignored. If false, all SendBuffers (also with zero size) are registered as ranks where messages are received from. The size is unknown, and communicated before.
changingSizeif true the size is communicated before every communication step. if useSizeFromSendBuffer==true and changingSize==true, the size is not communicated in the first step but in all following steps.
template<typename Rb , typename Sb >
size_t walberla::mpi::GenericBufferSystem< Rb, Sb >::size ( ) const
inline
template<typename Rb , typename Sb >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::sizeHasChanged ( bool  alwaysChangingSize = false)

Notifies that GenericBufferSystem that message sizes have changed ( and optionally are changing in all following steps)

Useful when setReceiverInfo was set such that GenericBufferSystem assumes constant message sizes for all steps. Can only be called if no communication is currently running.

Parameters
alwaysChangingSizeif true the message sizes is communicated in all following steps, if false only in the next step.
template<typename Rb , typename Sb >
void walberla::mpi::GenericBufferSystem< Rb, Sb >::startCommunication ( )
protected

Starts communication.

  • schedules receives and reserves space for MPI_Request vectors in the currentComm_ member
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
void walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::useIProbe ( const bool  use)
inline
template<typename Rb , typename Sb >
Rb * walberla::mpi::GenericBufferSystem< Rb, Sb >::waitForNext ( MPIRank &  fromRank)
protected

Helper function for iterator.

See documentation of AbstractCommunication::waitForNext()

Friends And Related Function Documentation

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
friend class GenericOpenMPBufferSystem< RecvBuffer_T, SendBuffer_T >
friend
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
friend class iterator
friend

Member Data Documentation

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
std::set< int > walberla::mpi::GenericBufferSystem< Rb, Sb >::activeTags_
staticprotected
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
int64_t walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::bytesReceived_ = 0
protected

number of bytes received during last communication

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
int64_t walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::bytesSent_ = 0
protected

number of bytes sent during last communication

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
bool walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::communicationRunning_
protected
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
internal::AbstractCommunication<RecvBuffer_T, SendBuffer_T>* walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::currentComm_
protected
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
internal::KnownSizeCommunication<RecvBuffer_T, SendBuffer_T> walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::knownSizeComm_
protected
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
internal::NoMPICommunication<RecvBuffer_T, SendBuffer_T> walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::noMPIComm_
protected
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
int64_t walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::numberOfReceives_ = 0
protected

number of communication partners during last receive

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
int64_t walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::numberOfSends_ = 0
protected

number of communication partners during last send

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
std::map<MPIRank, typename internal::AbstractCommunication<RecvBuffer_T, SendBuffer_T>::ReceiveInfo> walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::recvInfos_
protected

Info about the message to be received from a certain rank: information holds the buffer and, if known, the message size.

template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
std::map<MPIRank, SendInfo> walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::sendInfos_
protected
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
bool walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::sizeChangesEverytime_
protected
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
internal::UnknownSizeCommunication<RecvBuffer_T, SendBuffer_T> walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::unknownSizeComm_
protected
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
internal::UnknownSizeCommunicationIProbe<RecvBuffer_T, SendBuffer_T> walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::unknownSizeCommIProbe_
protected
template<typename RecvBuffer_T = RecvBuffer, typename SendBuffer_T = SendBuffer>
bool walberla::mpi::GenericBufferSystem< RecvBuffer_T, SendBuffer_T >::useIProbe_ = false
protected

switch betwenn IProbe and two message communication for varying size communication


The documentation for this class was generated from the following files: