TIBCO FTL Part of the TIBCO Messaging Suite. Quick Start Guide

Transcription

1 TIBCO FTL Part of the TIBCO Messaging Suite Quick Start Guide

2 The TIBCO Messaging Suite TIBCO FTL is part of the TIBCO Messaging Suite. It includes not only TIBCO FTL, but also TIBCO eftl (providing WebSocket support for mobile clients), TIBCO Enterprise Message Server (EMS), Apache Kafka (including support for Apache Kafka to TIBCO FTL connectivity), and MQTT (also including TIBCO FTL connectivity). All of these components are available to help TIBCO Messaging Suite users deploy the best messaging solution for each situation. All of these components can be used together to provide a powerful and complete messaging solution that extends from the data center, across your enterprise, out to mobile devices, and all the way to the most remote IoT device. About the Quick Start Guide This guide is focused on setting up, starting, and running various TIBCO FTL sample programs that demonstrate typical messaging functionality. You should be able to follow the examples and run through most of this guide in about 15 minutes. About FTL TIBCO FTL introduces a powerful new concept called the FTL Server. It provides a unified way to run all major FTL Services in a redundant, active-active manner that is easily deployed in cloud and container-based environments. Services like the Realm Server, persistence, bridges, groups, and others can be easily configured and quickly deployed to support FTL applications. Using the FTL Server model, applications can be rapidly deployed using a server-centric messaging model while keeping available all of the existing high performance, peer-to-peer connectivity options that TIBCO FTL has always provided. The Quick Start Guide includes examples of both types of models to demonstrate basic messaging as well as high performance messaging capabilities. Getting Started This document will help you get started by using TIBCO FTL samples to send and receive messages, use TIBCO FTL persistence for guaranteed message delivery, and help you collect some TIBCO FTL performance statistics using your own hardware and network. TIBCO FTL Quick Start for

3 Initial Configuration To accommodate TIBCO FTL potentially being installed in different locations, the samples in this document use the environment variable TIBCO_HOME to point to the directory where TIBCO FTL is installed. The default installation directory will be /opt/tibco on Linux and MacOS systems and C:\TIBCO on Windows systems. Set this environment variable and make sure it is available for other programs to access. On Linux and MacOS, use the export command to do this: $ export TIBCO_HOME=/opt/tibco On Windows, use the setx command: C:\> setx TIBCO_HOME C:\tibco For convenience purposes, we ll use the term FTL directory when referring to the base TIBCO FTL installation directory (e.g. $TIBCO_HOME/ftl/release_num where release_num is the version number of the TIBCO FTL release). NOTE: In most of this document, we ll use Linux-style commands but if you are using the Windows distribution of TIBCO FTL, you ll find the same scripts with the.bat suffix. For example, if the document refers to a script called setup, you ll find the script setup.bat in your Windows distribution. The examples will also using Linux-style paths so if you are using Windows, you ll want to replace forward slashes with backward slashes and appropriately change the syntax when referring to environment variables from their Linux-style (such as $TIBCO_HOME) to the corresponding Windows format (%TIBCO_HOME% for example). Setting up your PATH for easier execution Whether you are using the Linux, MacOS, or Windows TIBCO FTL distributions, running the examples will be easier if you add the appropriate TIBCO FTL bin and script directories to your PATH. It will simplify running many of the samples and scripts in this document because you won t have to specify full paths or change to different directories to run the various TIBCO FTL services or samples. To do this, you can add the following directories to your PATH: $TIBCO_HOME/ftl/release_num/bin $TIBCO_HOME/ftl/release_num/samples/bin $TIBCO_HOME/ftl/release_num/samples/scripts TIBCO FTL Quick Start for

4 There are Java versions of all of the various example applications located in the $TIBCO_HOME/ftl/release_num/samples/src/java directory. The Java samples require version 1.8 or later of the JRE (Java Runtime Environment); your PATH should also include the appropriate directories to access the JRE if you want to run the Java-based samples. Initializing the TIBCO FTL environment Before running any of the samples or the servers, you need to initialize the TIBCO FTL environment; the quickest way to do this is by running the setup script from the TIBCO FTL package. If you are opening multiple command windows, you will need do this in each command window in order to get the TIBCO FTL environment properly initialized. Navigate to the samples directory in the TIBCO FTL directory (e.g. $TIBCO_HOME/ftl/release_num/samples) and launch the setup script from a command line window. $../setup NOTE: On Linux and MacOS, be sure to include the preceding. ; this will make sure that the appropriate environment variables are set in your current shell. If you execute the setup script without the leading., the changes will not take effect in your current shell. On Windows, all you need to do is run the setup batch file. Starting and Stopping the FTL Server The FTL Server is the primary component which provides TIBCO FTL services and needs to be running before you can run any of the sample applications (or before you run any TIBCO FTL-based application for that matter). One of the key services provided by the FTL Server is the FTL realm service which serves as a repository of the configuration information needed by TIBCO FTL clients to communicate amongst themselves (all cooperating TIBCO FTL applications are part of a specific FTL realm which defines the various applications and connectivity parameters which are used to connect TIBCO FTL-based applications so that messages flow between them in the expected manner). To support the sample applications described in this guide, the TIBCO FTL distribution includes appropriate pre-defined configurations for the FTL Server as well as the FTL realm. The ftlstart command provided will load the configuration information needed to run the various sample applications. TIBCO FTL Quick Start for

5 As you gain more experience with TIBCO FTL, you ll likely want to modify the configuration to try other TIBCO FTL settings and options (such as using different transports to connect applications); in this case, you ll find the configuration provided to be a useful starting point. Starting the FTL Server To start the FTL Server, use the ftlstart command located in the samples/scripts directory. In this case, we ll start the simplest configuration; a single, non-redundant FTL Server running on the local system at a specific port: $ ftlstart ftls_1@localhost:8080 NOTE: The above example will start a single FTL Server named ftls_1 available at localhost port The FTL Server configuration used by ftlstart already has a the configuration for an FTL Server named ftls_1 so be sure to use that name. Any output from the FTL Server and any TIBCO FTL services for this FTL Server will be saved in the $HOME/ftl-server/ftls_1 directory. The ftlstart script will support either one or three three server strings on the command line (in addition to ftls_1, the configuration includes pre-defined FTL Servers ftls_2 and ftls_3 ). For running the examples in this Quick Start Guide, we re running just a single FTL Server but in actual production use you ll typically run three or more servers (usually in odd numbers to support the quorum-consensus model; see the TIBCO FTL Concepts document mentioned later for more details about this type of operation). Confirming the FTL Server is running If the FTL Server is running, you can point a browser to the appropriate host and port specified when you started the FTL Server ( if you followed the example above) and it will display realm configuration settings as well as status information. If you started more than one FTL Server, you can use any of the host and port settings you specified on the command line when you ran ftlstart to access the configuration and status information for the entire FTL realm. If you are not able to get a connection from your web browser, the FTL Server and it s associated realm service are likely not running; check for any error messages or warnings that were displayed at the time you started the FTL Server as well as looking in the appropriate $HOME/ftl-server/server-name directory. TIBCO FTL Quick Start for

6 NOTE: The most common problem when starting the FTL Server is the network port specified already being in use by another process (the examples in this document use port number 8080 but there is no guarantee that this port is actually available on any given system). If this is the case, you ll see an error message in the log output including the string Address already in use. You can specify any available port when starting the FTL Server so long as you specify the same host and port when running the sample applications. So if the network port specified in this guide is not available, it s perfectly OK to choose a different port, start your FTL Server using that port, and use that port when running the sample applications. Stopping the FTL Server After you are done running the samples, you can stop any running TIBCO FTL Servers from the command line by using the ftlstop script. NOTE: Run the ftlstop script with no options for usage information for the different types of configuration supported and how to stop them. In general, you ll want to use the same command line options as specified when you started the FTL Server(s) with the ftlstart script when you stop them using the ftlstop script. Running the samples As part of this Quick Start Guide, we ll cover three common messaging scenarios: Basic Publish-Subscribe Publish-Subscribe with Persistence Performance Measurement (messaging throughput and latency between two applications) Be sure you ve started the FTL Server as described above before running any of the samples; if the FTL Server is not running, the samples will wait and eventually time out with an error. If you changed the port the FTL Server is using, change the port numbers in the commands below to match the port you are actually using. If you are running more than one FTL Server, you can specific any one of the host:port values for the FTL servers on the command line or specify all of them on the command line separated by the character ( localhost:8080 localhost:9090 localhost:10100 for example; in this case, the application will choose one of the FTL Servers to connect to and automatically fall over to another in the event of no response or a disconnection). TIBCO FTL Quick Start for

7 The easiest way to run the various samples is to open multiple command windows and run different applications in different windows. That way, you ll be able to see the output from each application individually and get a clearer picture of what each sample application is doing. Be sure to source the setup file in each window to ensure that the TIBCO FTL environment is appropriately setup. Some of the sections relating to the sample applications will include one or more Going further... subsection; you can skip these areas for now if you want to move quickly through this document and just see the basic TIBCO FTL functionality available but they typically cover some additional capabilities that you may find useful or interesting. Feel free to revisit these sections at a later time to explore more TIBCO FTL functionality using the sample applications. Basic Publish-Subscribe sample In this example, we ll use the tibsendex sample application as a publisher to send messages which will be received by the subscriber in the tibrecvex sample application. In this example, we re demonstrating non-persistent messaging; a subscriber will receive any messages published by associated publishers as and when they are published. If a publisher publishes a message and there are no associated subscribers (or a specific subscriber has not yet been started), those messages will not be received by those subscribers. In a few moments, we ll also demonstrate persistent messaging which allows subscribers to receive messages published even when they were not running, but for now, we ll just focus on immediate message delivery. Open two command windows, source the setup file in each, and run tibrecvex sample application in one of the windows: $ tibrecvex c 10 The -c 10 command line option specifies that the receiver exit after receiving 10 messages. You ll see a final message waiting for message(s) displayed when the subscriber application is ready; at this point, we re ready to receive messages sent by associated publishers. In the other window, run the tibsendex sample application: $ tibsendex c 10 TIBCO FTL Quick Start for

8 As you probably expected, the -c 10 command line option specifies that the sender should exit after sending 10 messages. What did I just see? In the window running tibsendex, you ll see a series of output statements displayed as each message is published../tibsendex TIBCO FTL Version Invoked as:./tibsendex -c 10 sending message 1 sending message 2 sending message 3 sending message 4 sending message 5 sending message 6 sending message 7 sending message 8 sending message 9 sending message 10 To make it easy to follow what s happening, the messages are sent with a 1000ms delay between each message. You can change the number of messages sent using the -c command line option as well as the delay between messages using the -d option. Using the -? option will give you a full list of the available command line options which will give you control of a wide range oftibco FTL options and capabilities, but for now, if you want to change the behavior of tibsendex, stick with the -c and -d. In the window running tibrecvex, you ll see a series of output statements displayed as each message is received. Since both the subscriber and the publisher are running at the same time, the output relating to a message being published by tibsendex will be shown pretty much immediately as being received by tibrecvex../tibrecvex TIBCO FTL Version Invoked as:./tibrecvex -c 10 waiting for message(s) received message 1 message: type long: 1 type string: 'tibsend-endpoint' type opaque: size 6 data: 'opaque' received message 2 message: type long: 2 type string: 'tibsend-endpoint' type opaque: size 6 data: 'opaque' received message 3 TIBCO FTL Quick Start for

9 message: type long: 3 type string: 'tibsend-endpoint' type opaque: size 6 data: 'opaque'... some output not shown... type long: 9 type string: 'tibsend-endpoint' type opaque: size 6 data: 'opaque' received message 10 message: type long: 10 type string: 'tibsend-endpoint' type opaque: size 6 data: 'opaque' For each published message, you ll see 5 lines of output: The cumulative count of the number of messages received thus far (this count is maintained by the subscriber so you ll see this number incremented for each message received so if you have multiple publishers, you ll see it incremented each time a message is received giving you a running count of received messages) A header for the message contents ( message: ) The message contents on a field by field basis type long the message count added by the publisher (since this value is maintained and populated by the publisher, you ll see individual messages containing the same number coming from each publisher) type string the FTL endpoint name the publisher used to send the message type opaque six characters spelling out opaque in 7-bit ASCII Note: The information preceding the field content displays the type of data contained in a specific field in the message (the messages sent by tibsendex have three fields in them as shown above). While FTL can send completely arbitrary message content (typically sent as an opaque field type), the use of typed message fields gives the ability to do content-based addressing which is a very powerful feature in TIBCO FTL which we ll cover in a bit. Going Further Multiple Publishers and/or Subscribers In the example above, we used a single publisher and a single subscriber; feel free to run more of each to see multiple publisher and subscriber interactions. Try increasing the count and changing the delay values on tibsendex to see streams from multiple publishers arriving at a specific subscriber at different cadences. You can watch the TIBCO FTL Quick Start for

10 type long field to see message count from a specific publisher within an aggregated message stream Going Further Using Content Matchers One of the most powerful capabilities of TIBCO FTL is the ability to use the message content itself to control the flow and delivery of messages (referred to as Contentbased Addressing ). Content matchers can be used by subscribers to match specific messages using the actual content of the message, such as the existence of a specific named field (or lack of), all the way down to the contents of a specific named field. You can use the sample applications to see this behavior with tibrecvex by referencing the fields and field content in the message the tibsendex application is sending. For example, the command line below will match only one of the ten messages sent by tibsendex (the specific message where the My-Long field contains an integer value of 6): $ tibrecvex -c 10 m '{"My-Long":6}' NOTE: If you are using Windows, the command line syntax for the matcher is a bit different due to how the characters specifying the matcher are parsed at the command line; specify "{\"My-Long\":6}" (including the outside double quotes). If you re-run the publisher, you ll see publisher output messages sending message but you will only see a single message from the publisher at the subscriber using these options as it only actually receives the message where the My-Long field contains the value 6. You ll have to run the publisher 10 times to cause the subscriber to exit because the content matcher limited the number of received messages to only 1 of the 10 sent by the publisher (you can of course kill the receiver to end it at any time). You can use a content matcher to reduce the message processing code in your subscribing application by having your subscriber only receive messages it is actually interested in so you don t have to include the code to programmatically skip messages your application is not interested in. Going even further, content matchers from subscribers can be automatically shared all the way back to the publishing application and can actually reduce messaging traffic by not even sending messages to specific subscribers which they have indicated they are not interested in receiving (this can depend on the specific protocol used between the publisher and the subscriber but this can be a very powerful optimization capability to reduce message volume and the corresponding network traffic). TIBCO FTL Quick Start for

11 To help better understand Content-based Addressing, check out the video covering this exact topic at the TIBCO FTL Community site Persistent Publish-Subscribe sample In this example, we ll again use the tibsendex sample application acting as a publisher and tibrecvex as a subscriber but we ll include some additional steps and command line options to use persistent message delivery. In this mode, the publishers and subscribers are completely de-coupled and subscribers can recover messages which may have gotten lost between the publisher and the subscriber (due to network failure, resource issues anywhere along the delivery path, or any other failure). Open two command windows, source the setup file in each (or you can just keep using the windows you d used in the previous examples if they are still open). In one of the command windows, start the tibrecvex sample application using a few extra options: the -e option to tell the application to use a specific application endpoint that has been configured for persistence and the -d option which specifies a durable name (used to express interest in a particular message stream by a subscriber see the TIBCO FTL documentation for a complete explanation of how to use persistence including durables, stores, and clusters). $ tibrecvex -c 10 -d mydr -e tibrecv-persistentendpoint In the other command window, start the tibsendex sample application using the -e option as well. Note that we ve increased the number of messages sent from 10 to 20. $ tibsendex -c 20 -e tibsend-persistentendpoint You should see the tibrecvex application receive 10 messages and then exit but the tibsendex application continues to send until it has sent all 20 messages. So what about those 10 additional messages, are they lost? Not with persistence. They ve been retained by an FTL store which is the key component in TIBCO FTL persistence. If we run the tibrecvex application again using the same command line used above, we ll receive the remaining 10 messages even if the tibsendex application has completed sending all 20 messages and exited. Depending on the timing of the sending of the messages and the running of the receiver, you may see messages coming in very quickly at first with messages coming in at the slower cadence of the sender after this initial burst. What you are seeing is the subscriber catching up to the publisher after which you ll see the TIBCO FTL Quick Start for

12 message cadence slow down to reflect the actual sending rate of the publisher (if the publisher is still publishing). With persistence, the receiver was able to receive previously sent messages very quickly and once those have been delivered, it will receive subsequent messages in real-time. What happens if the FTL Server fails while messages are being published using persistence? Since we re only running a single FTL Server, we ll encounter message loss if we lose the FTL Server (or the system it s running on or the network connections to it, etc.) To increase reliability in production scenarios, you ll typically run multiple FTL Servers across multiple systems in order to provide redundancy for the persistence service itself in the event of a failure of a system, the network, or even the FTL Servers themselves; none of which will comprisie your message delivery. Going Further More Persistence Scenarios If you want to see how the persistence service ensures message delivery across subscriber failures and restarts, you can kill and restart the tibrecvex sample application before all of the messages are received and restart it to receive the remaining messages. You probably want to increase the message count to a significantly higher number to give you a little more time to kill the subscriber application before all of the messages have been delivered to the subscriber in order to more easily see this in action. Running the Performance Tests The TIBCO FTL product comes with a set of sample applications that can be used to measure the latency and the throughput on your own hardware. These samples can be run with any of the transports (e.g. shared memory, TCP, multicast, RUDP, etc.) but by default they are run using the shared memory transport for best performance. Before running the samples, please make sure that the FTL Server has been started and running as explained in the previous sections. Latency Tests The latency of a messaging system is a measure of how quickly a given message can be delivered between applications. The tiblatsend and tiblatrecv applications work together to provide a latency value calculated by averaging the time needed to send 5 million messages from the sender to the receiver. Once you have the FTL Server running, start the tiblatrecv application in one command window: TIBCO FTL Quick Start for

13 $ tiblatrecv Start the tiblatsend application in another command window: $ tiblatsend You ll see the results in the tiblatsend window giving you a summary of the total elapsed message delivery time, the number of messages sent, and an average representing the per message latency. Here is an example of the output: tiblatsend TIBCO FTL Version Invoked as: tiblatsend Calibrating tsc timer... done. CPU speed estimated to be 2.20E+09 Hz Sending messages with payload size 16 Sampling latency every messages. Total time: 4.57E+00 sec. for messages One way latency: E-09 sec. You can control the number of messages sent using the -count option as well as change the size of the messages sent using the -size option to simulate your specific messaging requirements. Use the -help option to see the compete set of command line options. Throughput Tests The throughput of a messaging system provides a measurement of the amount of data that can be delivered between applications over a period of time; typically when aiming for high throughput, you ll want to send fewer but larger messages to reduce the impact of permessage overhead on your messaging throughput. The tibthrusend and tibthrurecv applications work together to provide a throughput value by measuring the time needed to send 5 million messages from the sender to the receiver. Once you have the FTL Server running, start the tibthrurecv application in one command window: $ tibthrurecv Start the tibthrusend application in another command window: $ tibthrusend You ll see the results in the tibthrusend window giving you a summary of the number of messages sent (together with the number of batches and the batch size which can be set TIBCO FTL Quick Start for

14 using the -batchsize command line option), the total send and receive times, and the aggregate message send and receive rates. Typical output from tibthrusend would be: tibthrusend TIBCO FTL Version Invoked as: tibthrusend Calibrating tsc timer... done. CPU speed estimated to be 2.20E+09 Hz Sender Report: Requested messages. Sending messages (50000 batches of 100) with payload size 16 Sent 5.00E+06 messages in 2.92E+00 seconds. (1.71E+06 msgs/sec) Receiver Report: Received 5.00E+06 messages in 2.92E+00 seconds. (1.71E+06 messages per second) Received 80.00E+06 bytes in 2.92E+00 seconds. (27.38E+06 bytes per second) Messages per callback: min/max/avg/dev: 1.00E+00 / E+00 / 48.34E+00 / 29.37E+00 Based on this run, the sender sent all 5 million messages in 2.92 seconds (at an average rate of 1.72 million messages per second) and the receiver received these messages in about the same amount of time (at an average rate of 27 million bytes per second). These are pretty typical numbers for mid-range laptop level system with numerous other applications running (probably very much like your system during the work day running your typical development tools together with other applications like browsers and clients) and of course your performance will be greatly impacted by the performance and load of the system you are running on. You can control the number of messages sent using tibthrusend with the -count option (you ll want to keep these values relatively large in order to get representative results) as well as using the -size option to specify the size of the messages in bytes. The default size of 16 bytes is a relative small message and might not be representative of what your applications would be using; go ahead and try different values for --size to see it s impact on throughput using your hardware. Use the -help option to see the compete set of command line options for both tibthrusend and tibthrurecv. Documentation You can access the complete product documentation for TIBCO FTL at products/tibco-ftl-enterprise-edition-version where version is the 3 digit TIBCO FTL version number with each digit separated by dashes. For example, all of the documentation for the TIBCO FTL release is available at TIBCO FTL Quick Start for

15 You ll find the TIBCO FTL Concepts document a good starting point to understand the basics of TIBCO FTL in which will be useful as you get into the TIBCO FTL Administration and TIBCO FTL Development documentation. Where to go next At this point, you ve seen several TIBCO FTL sample programs providing basic messaging semantics including pub-sub and persistence for guaranteed message delivery as well as collecting some performance metrics using your own hardware. The source code for all of those samples (and more) are included as part of the TIBCO FTL distribution in the samples/src/c, samples/src/java, and samples/golang/src/tibco.com/ftlsample directories. Feel free to review the source code, make modifications, rebuild the applications, and run the modified versions. A good next step would be to start actually using TIBCO FTL to get familiar with the APIs and build some messaging applications. The TIBCO FTL Tutorials take a step-by-step approach towards building solid TIBCO FTL applications including describing various options and properties, exception handling, programming paradigms, and basic use of the UI for configuration and monitoring. The TIBCO FTL Tutorials are typically extended in each subsequent TIBCO FTL release; you ll always be able to get the latest version of the tutorials from the TIBCO FTL Community site at Getting support Support for the TIBCO FTL Community Edition is available at the TIBCO Community portal for TIBCO FTL located at You ll find information about new releases, how-to articles, videos, and other information. You can also post questions or comments and get feedback directly from the TIBCO FTL team. TIBCO FTL Enterprise Edition customers have access to all of that plus TIBCO s awardwinning 24 hour support programs. In addition, Enterprise Edition customers have access to several additional FTL capabilities such as the comprehensive monitoring dashboards and high performance connectivity options like RDMA support. TIBCO FTL Quick Start for