From a888156194db46694bb47619f82c13561f642dd8 Mon Sep 17 00:00:00 2001 From: Naveena Dhougoda Date: Thu, 22 Oct 2020 13:29:38 +0100 Subject: [PATCH] updated readme --- .gitignore | 1 + data-store/diffusion-kafka-app-L1/README.md | 28 ++++++++++----------- data-store/diffusion-kafka-app-L2/README.md | 25 ++++++++---------- data-store/diffusion-kafka-app-L3/README.md | 2 -- 4 files changed, 25 insertions(+), 31 deletions(-) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..723ef36 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.idea \ No newline at end of file diff --git a/data-store/diffusion-kafka-app-L1/README.md b/data-store/diffusion-kafka-app-L1/README.md index 36fce2d..88c22f9 100644 --- a/data-store/diffusion-kafka-app-L1/README.md +++ b/data-store/diffusion-kafka-app-L1/README.md @@ -1,16 +1,13 @@ # diffusion-kafka-app-L1 -Introduction to Diffusion Real-Time Data Store through a simple application using [Diffusion](https://www.pushtechnology.com/product-overview) Cloud and Apache Kafka. +# Lesson 1: Publish messages from Diffusion to Kafka +**diffusion-kafka-app-L1** introduces the concept of our [Kafka Adapter](https://www.pushtechnology.com/wp-content/uploads/2020/08/Diffusion-Cloud-Kafka-adapter.pdf) and its configuration. +In this lesson, I will present how to configure Kafka adapter in cloud, to consume [topic data](https://docs.pushtechnology.com/docs/6.5.1/manual/html/introduction/overview/topics_data.html) from +Diffusion server and publish it to Kafka cluster in specific Kafka topic. -A set of simple projects, illustrating production and consumption of foreign exchange (fx) data streams to and from Kafka clusters and Diffusion Cloud instance via the use of our [Kafka Adapter](https://www.pushtechnology.com/wp-content/uploads/2020/08/Diffusion-Cloud-Kafka-adapter.pdf). - -These JavaScript code examples will help you publish fx data on real-time from a front end app to a Kafka cluster, consume from it and transform data on-the-fly via our powerful [Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html) feature. You can also use other programming languages from our [SDKs](https://docs.pushtechnology.com/#sdks), including iOS, Android, C, .NET, and more. - -# Lesson 1: Publish Topics to Kafka cluster -**diffusion-kafka-app-L1** introduces the concept of our [Kafka Adapter](https://www.pushtechnology.com/wp-content/uploads/2020/08/Diffusion-Cloud-Kafka-adapter.pdf) and its configuration, as well as how to Publish fx data to ['Topics' (data structures)](https://docs.pushtechnology.com/docs/6.5.1/manual/html/introduction/overview/topics_data.html). In Diffusion, data is stored and distributed through Topics (similar to Kafka). - -# APIs used in this application +# **Step 1: Setup Diffusion publisher** +## APIs used in this application ## **Step 1: Connect to Diffusion** ### [diffusion.connect](https://docs.pushtechnology.com/docs/6.5.1/js/globals.html#connect) > [*create your host*](https://management.ad.diffusion.cloud/) ```js @@ -25,8 +22,9 @@ diffusion.connect({ session.topics.add(_fxTopic, diffusion.topics.TopicType.JSON); ``` ### Go to: [Diffusion Cloud > Manage Service > Console > Topics](https://management.ad.diffusion.cloud/#!/login) -We are seeting up `_fxTopic` with the topic path: `diffusion.fx` +We are setting up `_fxTopic` with the topic path: `diffusion.fx` ![](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L1/images/topics.png) + ## **Step 3: Update a Topic** ### [session.topicUpdate.set](https://docs.pushtechnology.com/docs/6.5.1/js/interfaces/topicupdate.html#set) ```js @@ -50,16 +48,18 @@ session.topicUpdate.createUpdateStream(_fxTopic, diffusion.datatypes.json()).set offer : offer.toFixed(2) }); ``` -## **Step 4: Kafka Adapter Configuration > Broadcast to Kafka** + +# **Step 2: Configure Kafka Adapter in Cloud** +## Kafka Adapter Configuration > Broadcast to Kafka ### Go to: [Diffusion Cloud > Manage Service > Adapters > Kafka Adapter](https://management.ad.diffusion.cloud/#!/login) [![Kafka Adapter Video](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L1/images/adapter.png)](https://www.pushtechnology.com/blog/how-to-build-a-real-time-messaging-app-using-diffusion/) ```js -Adpapters > Kafka Adapter > Broadcast_to_Kafka Config: +Adapters > Kafka Adapter > Broadcast_to_Kafka Config: Broker > connect to you Kafka cluster > (eg: "kafka-plain.preprod-demo.pushtechnology.com:9094") Diffusion service credentials > admin, password, > (use the "Security" tab to create a user or admin account) - Diffusion Topic > the data source to broadcast to Kafka cluster > (eg: "diffusion.fx") - Kafka Topic > the destination topic at your Kafka cluster > (eg: "kafka.fx") + Diffusion Topic > the Diffusion topic selector, from which messages are to be broadcasted to Kafka cluster > (eg: "diffusion.fx") + Kafka Topic > the destination Kafka topic at your Kafka cluster > (eg: "kafka.fx") ``` diff --git a/data-store/diffusion-kafka-app-L2/README.md b/data-store/diffusion-kafka-app-L2/README.md index 98e5f41..1e52a41 100644 --- a/data-store/diffusion-kafka-app-L2/README.md +++ b/data-store/diffusion-kafka-app-L2/README.md @@ -1,31 +1,26 @@ # diffusion-kafka-app-L2 -Introduction to Diffusion Real-Time Data Store through a simple application using [Diffusion](https://www.pushtechnology.com/product-overview) Cloud and Apache Kafka. - -A set of simple projects, illustrating production and consumption of foreign exchange (fx) data streams to and from Kafka clusters and Diffusion Cloud instance via the use of our [Kafka Adapter](https://www.pushtechnology.com/wp-content/uploads/2020/08/Diffusion-Cloud-Kafka-adapter.pdf). - -These JavaScript code examples will help you publish fx data on real-time from a front end app to a Kafka cluster, consume from it and transform data on-the-fly via our powerful [Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html) feature. You can also use other programming languages from our [SDKs](https://docs.pushtechnology.com/#sdks), including iOS, Android, C, .NET, and more. - # Lesson 2: Ingest Kafka Topics (firehose) and Create Topic Views -**diffusion-kafka-app-L2** introduces the concept of [Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html), a dynamic mechanism to map part of a server's [Topic Tree](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_tree.html) (data structure) to another. This enables real-time data transformation before sending it to subscribed clients as well as to create dynamic data models based on on-the-fly data (eg: Kafka firehose data). +**diffusion-kafka-app-L2** introduces the concept of [Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html), + a dynamic mechanism to map part of a server's [Topic Tree](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_tree.html) to another. + This enables real-time data transformation before sending it to subscribed clients as well as to create dynamic data models based on on-the-fly data (eg: Kafka firehose data). -# APIs used in this application - -## Step 1: Kafka Adapter Configuration > Ingest from Kafka +## Step 1: Configure Kafka Adapter in cloud ### Go to: [Diffusion Cloud > Manage Service > Adapters > Kafka Adapter](https://management.ad.diffusion.cloud/#!/login) [![Kafka Adapter Video](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L2/images/ingest.png)](https://www.pushtechnology.com/blog/how-to-build-a-real-time-messaging-app-using-diffusion/) ```js -Adpapters > Kafka Adapter > Ingest_from_Kafka Config: +Adapters > Kafka Adapter > Ingest_from_Kafka Config: Bootstrap Server > connect to you Kafka cluster > (eg: "kafka-plain.preprod-demo.pushtechnology.com:9094") Diffusion service credentials > admin, password, > (use the "Security" tab to create a user or admin account) Kafka Topic subscription > the source topic from your Kafka cluster > (eg: "kafka.firehose.fx") Kafka Topic value type > we are using JSON but can be string, integer, byte, etc. ``` + ## Step 2: Check the Kafka stream is ingested ### Go to: [Diffusion Cloud > Manage Service > Console > Topics](https://management.ad.diffusion.cloud/#!/login) -We can see the `kafka.firehose.fx` stream from Kafka cluster (we set up on previous step) is now being ingested by Diffusion with the topic path: `kafka.firehose.fx` - +We can see the message from `kafka.firehose.fx` Kafka topic (we set up on previous step) is now being published to Diffusion topic path: `kafka.firehose.fx` +If there are no messages, it might be because the `kafka.firehose.fx` topic has not received any updates from Kafka. ![](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L2/images/kafka%20firehose.png) ## Step 3: Create a Topic View using [Source value directives](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html) @@ -37,8 +32,8 @@ We are going to map `kafka.firehose.fx` stream (we set up on previous step) to a ![](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L2/images/topic%20views.png) -## Step 4: Check the Topic View is multiplexing the Kafka firehose -As new values are coming in from the Kafka firehose, Diffusion is multiplexing and publishing the currency pairs on real-time. +## Step 4: Check reference topics are being updated +As new values are coming in from the Kafka firehose, the reference topics created by topic views are updated with the currency pairs on real-time. **Note:** The topic path will dynamically change as new currency pair values come in. ### Go to: [Diffusion Cloud > Manage Service > Console > Topics](https://management.ad.diffusion.cloud/#!/login) diff --git a/data-store/diffusion-kafka-app-L3/README.md b/data-store/diffusion-kafka-app-L3/README.md index 96098f8..edcc520 100644 --- a/data-store/diffusion-kafka-app-L3/README.md +++ b/data-store/diffusion-kafka-app-L3/README.md @@ -9,8 +9,6 @@ These JavaScript code examples will help you publish fx data on real-time from a # Lesson 3: Subscribe to Kafka Topic Views **diffusion-kafka-app-L3** introduces the concept of [Security](https://docs.pushtechnology.com/docs/6.5.1/manual/html/designguide/security/c_security.html) and topic [path permissions](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/security/permissions_reference.html) for fine-grained security management of your data structure. It also shows how to Subscribe to fx data using [Diffusion Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html) in order to consume what you need, not all the Kafka stream. -# APIs used in this application - ## Step 1: Kafka Adapter Configuration > Ingest from Kafka ### Go to: [Diffusion Cloud > Manage Service > Adapters > Kafka Adapter](https://management.ad.diffusion.cloud/#!/login) [![Kafka Adapter Video](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L2/images/ingest.png)](https://www.pushtechnology.com/blog/how-to-build-a-real-time-messaging-app-using-diffusion/)