Skip to content

updated readme #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
naveenad wants to merge 1 commit into
base: master
Choose a base branch
from
Open

updated readme #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.idea
28 changes: 14 additions & 14 deletions data-store/diffusion-kafka-app-L1/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
# diffusion-kafka-app-L1

Introduction to Diffusion Real-Time Data Store through a simple application using [Diffusion](https://www.pushtechnology.com/product-overview) Cloud and Apache Kafka.
# Lesson 1: Publish messages from Diffusion to Kafka
**diffusion-kafka-app-L1** introduces the concept of our [Kafka Adapter](https://www.pushtechnology.com/wp-content/uploads/2020/08/Diffusion-Cloud-Kafka-adapter.pdf) and its configuration.
In this lesson, I will present how to configure Kafka adapter in cloud, to consume [topic data](https://docs.pushtechnology.com/docs/6.5.1/manual/html/introduction/overview/topics_data.html) from
Diffusion server and publish it to Kafka cluster in specific Kafka topic.

A set of simple projects, illustrating production and consumption of foreign exchange (fx) data streams to and from Kafka clusters and Diffusion Cloud instance via the use of our [Kafka Adapter](https://www.pushtechnology.com/wp-content/uploads/2020/08/Diffusion-Cloud-Kafka-adapter.pdf).

These JavaScript code examples will help you publish fx data on real-time from a front end app to a Kafka cluster, consume from it and transform data on-the-fly via our powerful [Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html) feature. You can also use other programming languages from our [SDKs](https://docs.pushtechnology.com/#sdks), including iOS, Android, C, .NET, and more.

# Lesson 1: Publish Topics to Kafka cluster
**diffusion-kafka-app-L1** introduces the concept of our [Kafka Adapter](https://www.pushtechnology.com/wp-content/uploads/2020/08/Diffusion-Cloud-Kafka-adapter.pdf) and its configuration, as well as how to Publish fx data to ['Topics' (data structures)](https://docs.pushtechnology.com/docs/6.5.1/manual/html/introduction/overview/topics_data.html). In Diffusion, data is stored and distributed through Topics (similar to Kafka).

# APIs used in this application

# **Step 1: Setup Diffusion publisher**
## APIs used in this application
## **Step 1: Connect to Diffusion**
### [diffusion.connect](https://docs.pushtechnology.com/docs/6.5.1/js/globals.html#connect) > [*create your host*](https://management.ad.diffusion.cloud/)
```js
Expand All @@ -25,8 +22,9 @@ diffusion.connect({
session.topics.add(_fxTopic, diffusion.topics.TopicType.JSON);
```
### Go to: [Diffusion Cloud > Manage Service > Console > Topics](https://management.ad.diffusion.cloud/#!/login)
We are seeting up `_fxTopic` with the topic path: `diffusion.fx`
We are setting up `_fxTopic` with the topic path: `diffusion.fx`
![](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L1/images/topics.png)

## **Step 3: Update a Topic**
### [session.topicUpdate.set](https://docs.pushtechnology.com/docs/6.5.1/js/interfaces/topicupdate.html#set)
```js
Expand All @@ -50,16 +48,18 @@ session.topicUpdate.createUpdateStream(_fxTopic, diffusion.datatypes.json()).set
offer : offer.toFixed(2)
});
```
## **Step 4: Kafka Adapter Configuration > Broadcast to Kafka**

# **Step 2: Configure Kafka Adapter in Cloud**
## Kafka Adapter Configuration > Broadcast to Kafka
### Go to: [Diffusion Cloud > Manage Service > Adapters > Kafka Adapter](https://management.ad.diffusion.cloud/#!/login)
[![Kafka Adapter Video](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L1/images/adapter.png)](https://www.pushtechnology.com/blog/how-to-build-a-real-time-messaging-app-using-diffusion/)
```js
Adpapters > Kafka Adapter > Broadcast_to_Kafka Config:
Adapters > Kafka Adapter > Broadcast_to_Kafka Config:

Broker > connect to you Kafka cluster > (eg: "kafka-plain.preprod-demo.pushtechnology.com:9094")
Diffusion service credentials > admin, password, > (use the "Security" tab to create a user or admin account)
Diffusion Topic > the data source to broadcast to Kafka cluster > (eg: "diffusion.fx")
Kafka Topic > the destination topic at your Kafka cluster > (eg: "kafka.fx")
Diffusion Topic > the Diffusion topic selector, from which messages are to be broadcasted to Kafka cluster > (eg: "diffusion.fx")
Kafka Topic > the destination Kafka topic at your Kafka cluster > (eg: "kafka.fx")

```

Expand Down
25 changes: 10 additions & 15 deletions data-store/diffusion-kafka-app-L2/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,31 +1,26 @@
# diffusion-kafka-app-L2

Introduction to Diffusion Real-Time Data Store through a simple application using [Diffusion](https://www.pushtechnology.com/product-overview) Cloud and Apache Kafka.

A set of simple projects, illustrating production and consumption of foreign exchange (fx) data streams to and from Kafka clusters and Diffusion Cloud instance via the use of our [Kafka Adapter](https://www.pushtechnology.com/wp-content/uploads/2020/08/Diffusion-Cloud-Kafka-adapter.pdf).

These JavaScript code examples will help you publish fx data on real-time from a front end app to a Kafka cluster, consume from it and transform data on-the-fly via our powerful [Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html) feature. You can also use other programming languages from our [SDKs](https://docs.pushtechnology.com/#sdks), including iOS, Android, C, .NET, and more.

# Lesson 2: Ingest Kafka Topics (firehose) and Create Topic Views
**diffusion-kafka-app-L2** introduces the concept of [Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html), a dynamic mechanism to map part of a server's [Topic Tree](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_tree.html) (data structure) to another. This enables real-time data transformation before sending it to subscribed clients as well as to create dynamic data models based on on-the-fly data (eg: Kafka firehose data).
**diffusion-kafka-app-L2** introduces the concept of [Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html),
a dynamic mechanism to map part of a server's [Topic Tree](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_tree.html) to another.
This enables real-time data transformation before sending it to subscribed clients as well as to create dynamic data models based on on-the-fly data (eg: Kafka firehose data).

# APIs used in this application

## Step 1: Kafka Adapter Configuration > Ingest from Kafka
## Step 1: Configure Kafka Adapter in cloud
### Go to: [Diffusion Cloud > Manage Service > Adapters > Kafka Adapter](https://management.ad.diffusion.cloud/#!/login)
[![Kafka Adapter Video](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L2/images/ingest.png)](https://www.pushtechnology.com/blog/how-to-build-a-real-time-messaging-app-using-diffusion/)
```js
Adpapters > Kafka Adapter > Ingest_from_Kafka Config:
Adapters > Kafka Adapter > Ingest_from_Kafka Config:

Bootstrap Server > connect to you Kafka cluster > (eg: "kafka-plain.preprod-demo.pushtechnology.com:9094")
Diffusion service credentials > admin, password, > (use the "Security" tab to create a user or admin account)
Kafka Topic subscription > the source topic from your Kafka cluster > (eg: "kafka.firehose.fx")
Kafka Topic value type > we are using JSON but can be string, integer, byte, etc.
```

## Step 2: Check the Kafka stream is ingested
### Go to: [Diffusion Cloud > Manage Service > Console > Topics](https://management.ad.diffusion.cloud/#!/login)
We can see the `kafka.firehose.fx` stream from Kafka cluster (we set up on previous step) is now being ingested by Diffusion with the topic path: `kafka.firehose.fx`

We can see the message from `kafka.firehose.fx` Kafka topic (we set up on previous step) is now being published to Diffusion topic path: `kafka.firehose.fx`
If there are no messages, it might be because the `kafka.firehose.fx` topic has not received any updates from Kafka.
![](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L2/images/kafka%20firehose.png)

## Step 3: Create a Topic View using [Source value directives](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html)
Expand All @@ -37,8 +32,8 @@ We are going to map `kafka.firehose.fx` stream (we set up on previous step) to a

![](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L2/images/topic%20views.png)

## Step 4: Check the Topic View is multiplexing the Kafka firehose
Copy link
Author

@naveenad naveenad Oct 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not very clear. The term “multiplexing” might be confusing to users.

As new values are coming in from the Kafka firehose, Diffusion is multiplexing and publishing the currency pairs on real-time.
## Step 4: Check reference topics are being updated
As new values are coming in from the Kafka firehose, the reference topics created by topic views are updated with the currency pairs on real-time.

**Note:** The topic path will dynamically change as new currency pair values come in.
### Go to: [Diffusion Cloud > Manage Service > Console > Topics](https://management.ad.diffusion.cloud/#!/login)
Expand Down
2 changes: 0 additions & 2 deletions data-store/diffusion-kafka-app-L3/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ These JavaScript code examples will help you publish fx data on real-time from a
# Lesson 3: Subscribe to Kafka Topic Views
**diffusion-kafka-app-L3** introduces the concept of [Security](https://docs.pushtechnology.com/docs/6.5.1/manual/html/designguide/security/c_security.html) and topic [path permissions](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/security/permissions_reference.html) for fine-grained security management of your data structure. It also shows how to Subscribe to fx data using [Diffusion Topic Views](https://docs.pushtechnology.com/docs/6.5.2/manual/html/designguide/data/topictree/topic_views.html) in order to consume what you need, not all the Kafka stream.

# APIs used in this application
Copy link
Author

@naveenad naveenad Oct 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no APIs used for these steps.

Copy link
Contributor

@MatiasPush MatiasPush Oct 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


## Step 1: Kafka Adapter Configuration > Ingest from Kafka
### Go to: [Diffusion Cloud > Manage Service > Adapters > Kafka Adapter](https://management.ad.diffusion.cloud/#!/login)
[![Kafka Adapter Video](https://github.com/pushtechnology/tutorials/blob/master/data-store/diffusion-kafka-app-L2/images/ingest.png)](https://www.pushtechnology.com/blog/how-to-build-a-real-time-messaging-app-using-diffusion/)
Expand Down