/


    / (9 CONNECTED)
    

/Event

/

Connections require the use of both passive and active endpoints.
In order to establish a connection, an application must first create a
passive endpoint and associate it with an event queue. The event queue
will be used to report the connection management events. The application
then calls listen on the passive endpoint. A single passive endpoint can
be used to form multiple connections.
The connecting peer allocates an active endpoint, which is also
associated with an event queue. Connect is called on the active
endpoint, which results in sending a connection request (CONNREQ)
message to the passive endpoint. The CONNREQ event is inserted into
the passive endpoint’s event queue, where the listening application can
process it.
Upon processing the CONNREQ, the listening application will allocate
an active endpoint to use with the connection. The active endpoint is
bound with an event queue. Although the diagram shows the use of a
separate event queue, the active endpoint may use the same event queue
as used by the passive endpoint. Accept is called on the active endpoint
to finish forming the connection. It should be noted that the OFI accept
call is different than the accept call used by sockets. The differences
result from OFI supporting process direct I/O.
libfabric does not define the connection establishment protocol, but
does support a traditional three-way handshake used by many technologies.
After calling accept, a response is sent to the connecting active endpoint.
That response generates a CONNECTED event on the remote event queue. If a
three-way handshake is used, the remote endpoint will generate an
acknowledgment message that will generate a CONNECTED event for the accepting
endpoint. Regardless of the connection protocol, both the active and passive
sides of the connection will receive a CONNECTED event that signals that the
connection has been established.
## Connectionless Communications
Connectionless communication allows data transfers between active endpoints
without going through a connection setup process. The diagram below shows
the basic components needed to setup connection-less communication.
Connectionless communication setup differs from UDP sockets in that it
requires that the remote addresses be stored with libfabric.
    

1 insert_addr() 2 send() | | /Address <–3 lookup–> / Active

/ /

libfabric requires the addresses of peer endpoints be inserted into a local
addressing table, or address vector, before data transfers can be initiated
against the remote endpoint. Address vectors abstract fabric specific
addressing requirements and avoid long queuing delays on data transfers
when address resolution is needed. For example, IP addresses may need to be
resolved into Ethernet MAC addresses. Address vectors allow this resolution
to occur during application initialization time. libfabric does not define
how an address vector be implemented, only its conceptual model.
All connection-less endpoints that transfer data must be associated with an
address vector.
# Endpoints
At a low-level, endpoints are usually associated with a transmit context, or
queue, and a receive context, or queue.  Although the terms transmit and
receive queues are easier to understand, libfabric uses the terminology
context, since queue like behavior of acting as a FIFO (first-in, first-out)
is not guaranteed.  Transmit and receive contexts may be implemented using
hardware queues mapped directly into the process’s address space.  An endpoint
may be configured only to transmit or receive data.  Data transfer requests
are converted by the underlying provider into commands that are inserted into
hardware transmit and/or receive contexts.
Endpoints are also associated with completion queues. Completion queues are
used to report the completion of asynchronous data transfer operations.
## Shared Contexts
An advanced usage model allows for sharing resources among multiple endpoints.
The most common form of sharing is having multiple connected endpoints
make use of a single receive context.  This can reduce receive side buffering
requirements, allowing the number of connected endpoints that an application
can manage to scale to larger numbers.
# Data Transfers
Obviously, a primary goal of network communication is to transfer data between
processes running on different systems. In a similar way that the socket API
defines different data transfer semantics for TCP versus UDP sockets, that is,
streaming versus datagram messages, libfabric defines different types of data
transfers. However, unlike sockets, libfabric allows different semantics over
a single endpoint, even when communicating with the same peer.
libfabric uses separate API sets for the different data transfer semantics;
although, there are strong similarities between the API sets.  The differences
are the result of the parameters needed to invoke each type of data transfer.
## Message transfers
Message transfers are most similar to UDP datagram transfers, except that
transfers may be sent and received reliably.  Message transfers may also be
gigabytes in size, depending on the provider implementation.  The sender
requests that data be transferred as a single transport operation to a peer.
Even if the data is referenced using an I/O vector, it is treated as a single
logical unit or message.  The data is placed into a waiting receive buffer
at the peer, with the receive buffer usually chosen using FIFO ordering.
Note that even though receive buffers are selected using FIFO ordering, the
received messages may complete out of order.  This can occur as a result of
data between and within messages taking different paths through the network,
handling lost or retransmitted packets, etc.
Message transfers are usually invoked using API calls that contain the string
"send" or "recv".  As a result they may be referred to simply as sent or
received messages.
Message transfers involve the target process posting memory buffers to the
receive (Rx) context of its endpoint.  When a message arrives from the network,
a receive buffer is removed from the Rx context, and the data is copied from
the network into the receive buffer.  Messages are matched with posted receives
in the order that they are received.  Note that this may differ from the order
that messages are sent, depending on the transmit side's ordering semantics.
Conceptually, on the transmit side, messages are posted to a transmit (Tx)
context.  The network processes messages from the Tx context, packetizing
the data into outbound messages.  Although many implementations process the
Tx context in order (i.e. the Tx context is a true queue), ordering guarantees
specified through the libfabric API determine the actual processing order.  As
a general rule, the more relaxed an application is on its message and data
ordering, the more optimizations the networking software and hardware can
leverage, providing better performance.
## Tagged messages
Tagged messages are similar to message transfers except that the messages
carry one additional piece of information, a message tag.  Tags are application
defined values that are part of the message transfer protocol and are used to
route packets at the receiver.  At a high level, they are roughly similar to
message ids.  The difference is that tag values are set by the application,
may be any value, and duplicate tag values are allowed.
Each sent message carries a single tag value, which is used to select a receive
buffer into which the data is copied.  On the receiving side, message buffers
are also marked with a tag.  Messages that arrive from the network search
through the posted receive messages until a matching tag is found.
Tags are often used to identify virtual communication groups or roles.
In practice, message tags are typically divided into fields.  For example, the
upper 16 bits of the tag may indicate a virtual group, with the lower 16 bits
identifying the message purpose.  The tag message interface in libfabric is
designed around this usage model.  Each sent message carries exactly one tag
value, specified through the API.  At the receiver, buffers are associated
with both a tag value and a mask.  The mask is used as part of the buffer
matching process.  The mask is applied against the received tag value carried
in the sent message prior to checking the tag against the receive buffer.  For
example, the mask may indicate to ignore the lower 16-bits of a tag.  If
the resulting values match, then the tags are said to match.  The received
data is then placed into the matched buffer.
For performance reasons, the mask is specified as 'ignore' bits. Although
this is backwards from how many developers think of a mask (where the bits
that are valid would be set to 1), the definition ends up mapping well with
applications.  The actual operation performed when matching tags is:
    

send_tag | ignore == recv_tag | ignore

/* this is equivalent to: * send_tag & ~ignore == recv_tag & ~ignore */ ```

Tagged messages are equivalent of message transfers if a single tag value is used. But tagged messages require that the receiver perform a matching operation at the target, which can impact performance versus untagged messages.

RMA¶

RMA operations are architected such that they can require no processing by the CPU at the RMA target. NICs which offload transport functionality can perform RMA operations without impacting host processing. RMA write operations transmit data from the initiator to the target. The memory location where the data should be written is carried within the transport message itself, with verification checks at the target to prevent invalid access.

RMA read operations fetch data from the target system and transfer it back to the initiator of the request, where it is placed into memory. This too can be done without involving the host processor at the target system when the NIC supports transport offloading.

The advantage of RMA operations is that they decouple the processing of the peers. Data can be placed or fetched whenever the initiator is ready without necessarily impacting the peer process.

Because RMA operations allow a peer to directly access the memory of a process, additional protection mechanisms are used to prevent unintentional or unwanted access. RMA memory that is updated by a write operation or is fetched by a read operation must be registered for access with the correct permissions specified.

Atomic operations¶

Atomic transfers are used to read and update data located in remote memory regions in an atomic fashion. Conceptually, they are similar to local atomic operations of a similar nature (e.g. atomic increment, compare and swap, etc.). The benefit of atomic operations is they enable offloading basic arithmetic capabilities onto a NIC. Unlike other data transfer operations, which merely need to transfer bytes of data, atomics require knowledge of the format of the data being accessed.

A single atomic function operates across an array of data, applying an atomic operation to each entry. The atomicity of an operation is limited to a single data type or entry, however, not across the entire array. libfabric defines a wide variety of atomic operations across all common data types. However support for a given operation is dependent on the provider implementation.

Collective operations¶

In general, collective operations can be thought of as coordinated atomic operations between a set of peer endpoints, almost like a multicast atomic request. A single collective operation can result in data being collected from multiple peers, combined using a set of atomic primitives, and the results distributed to all peers. A collective operation is a group communication exchange. It involves multiple peers exchanging data with other peers participating in the collective call. Collective operations require close coordination by all participating members, and collective calls can strain the fabric, as well as local and remote data buffers.

Collective operations are an area of heavy research, with dedicated libraries focused almost exclusively on implementing collective operations efficiently. Such libraries are a specific target of libfabric. The main object of the libfabric collection APIs is to expose network acceleration features for implementing collectives to higher-level libraries and applications. It is recommended that applications needing collective communication target higher-level libraries, such as MPI, instead of using libfabric collective APIs for that purpose.

AUTHORS¶

OpenFabrics.