A step-by-step tutorial on building a live Twitter like search feed with React.JS and appbase.io.
This is Part — II of a series on building Twitter like live search feeds with different Javascript frameworks. You can check out the Part-I published on scotch.io where we used jQuery.
Ever tried Twitter Live Search and wondered how it works? On the surface, you search for a #hashtag or a keyword and Twitter shows you a live feed of results, with new tweets appearing continuously after the initial search results were rendered completely!
Image: Live twitter search for #GameOfThrones
How is Twitter able to return a dynamic result feed for any searched keyword? One way to implement a Twitter Search style live feed is to return the original search results from a database (SQL, ElasticSearch, Mongo etc.) and then have a separate feed component using a realtime channel (think socket.io).
Image: Architecture for a naive realtime feed
We can’t know for sure how Twitter internally implements the realtime feed, but the approach described above might be easy to get started but requires scaling two different system components, the database and the feed logic. It can also suffer from data consistency issues because of constant syncing that is required by this arranged. Performance wise, it would do okay with a O(MxN) time complexity (where M=data insertions into the DB per batch, N=channels with at least one active subscriber).
In this post, we will describe a scalable approach to building realtime feeds using Meetup’s RSVPs as the data source. We will store this data in appbase.io, which acts as a database (thanks to Elasticsearch) with a realtime feeds layer. For the backend, we will use a Node.JS worker to fetch Meetup’s RSVPs and insert them into an appbase.io app. On the frontend, we will use React.JS to create the feed UI and query appbase.io in realtime with different keyword filters.
#Enter meetupblast!
Meetupblast shows a live feed of meetup RSVPs searchable by cities and topics. Like Twitter Live Search, it returns a dynamic feed of RSVP data by the selected cities and topics. It’s a great way to find cool meetups other people are attending in your city.
(http://appbaseio-apps.github.io/meetupblast-react/)
You can try the live demo and access the code on github here or continue reading to see how we built this.
Key Ingredients
The meetupblast recipe can be broken into two key pieces: the backend worker and the user interface.
- Backend Worker
- We use Meetup’s RSVP stream endpoint to get realtime RSVPs.
- We then store this data in appbase.io, which provides us a convenient way to query both historical data and realtime feeds — we call it the streaming DB ;)
- User Interface
- Querying appbase.io for Meetup RSVP feeds by cities and topics.
- The UI / UX logic is entirely written in a React.JS frontend. And we use typeahead for displaying the search results, it’s a convenience library for building search interfaces from Twitter.
#Deep Dive
Now that we know what meetupblast does, let’s get to the thick of how the app works.
Backend Worker
Our backend worker is a simple Node.JS code that keeps running forever on a DigitalOcean droplet.
The worker consumes meetup RSVP data from their APIs
http.get(meetup_url, function(res) { res.on('data', function(chunk) { // called on new RSVPs var data = JSON.parse(chunk); meetup_data.push(data); // capture RSVPs in an array }); });
and then index each RSVP into appbase.io.
appbaseRef.index({ type: DATA_TABLE,// collection to store the data into body: meetup_data[counter] }).on('data', function(res) { console.log("successfully indexed one meetup RSVP"); })
Meetup provides us a JSON object for every RSVP. We then write this data into appbase.io as soon as it arrives. appbase.io is built as a streaming layer on ElasticSearch, and provides a live query interface for streaming JSON results.
An RSVP JSON data looks like this:
"visibility": "public", "response": "yes", "guests": 0, "member": { "member_id": 185034988, "photo": "http://photos1.meetupstatic.com/photos/member/d/0/0/0/thumb_248333248.jpeg", "member_name": "Wilma" }, "rsvp_id": 1566804180, "mtime": 1440266230993, "event": { "event_name": "Wednesday Badminton @ Flying Dragon", "event_id": "224809211", "time": 1440630000000, "event_url": "http://www.meetup.com/Richmond-Hill-Friends/events/224809211/" }, "group": { "group_topics": [ { "urlkey": "social", "topic_name": "Social" }, { "urlkey": "board-games", "topic_name": "Board Games" }, { "urlkey": "movie-nights", "topic_name": "Movie Nights" } ], "group_city": "Richmond Hill", "group_country": "ca", "group_id": 1676043, "group_name": "Richmond Hill Friends", "group_lon": -79.4, "group_urlname": "Richmond-Hill-Friends", "group_state": "ON", "group_lat": 43.84 }
The User Interface is a small frontend that queries appbase.io for realtime meetups based on specific cities and tags and displays them in a neat list.
Image: Building the live feed interface with appbase.io, typeahead.js and react.js
The final app directory structure will look as follows:
The codebase can be accessed at the meetupblast github repo. Or you can follow the remaining tutorial to see how we build it step by step.
#Step 0: Initial Project Setup
We use bower_components for managing library dependencies for bootstrap, appbase-js and typeahead.js. If you haven’t used it before, it’s a neat package manager for the web.
bower init /* follow the interactive setup to create bower.json */ bower install --save bootstrap bower install --save appbase-js bower install --save typeahead.js
We will use browserify and reactify, two node_modules along with gulp, a streaming build for transpiling all the React .jsx files we will soon be adding to our codebase into javascript.
npm init /* follow the interactive setup to create package.json */ npm install --save browserify npm install --save reactify npm install --save gulp (if you don't already have it)
If you haven’t used gulp before or would like to learn more about how the transpiling process works in greater depth, checkout this spot on tutorial by Tyler McGinnis on the topic.
Next, we will configure the gulpfile to read all our React files.
var browserify = require('browserify'); var gulp = require('gulp'); var source = require("vinyl-source-stream"); var reactify = require('reactify'); gulp.task('browserify', function() { var b = browserify({ entries: ['src/app.js'], debug: true }); b.transform(reactify); // use the reactify transform return b.bundle() .pipe(source('main.js')) .pipe(gulp.dest('./dist')); }); gulp.task('watch', function() { gulp.watch('src/*.js', ['browserify']); gulp.watch('src/*.jsx', ['browserify']); }); gulp.task('default', ['watch', 'browserify']);
A gulp file typically has one or more build tasks. We define the transpiling taskbrowserify, which reactifies all the .jsx files starting from src/app.js into a single dist/main.js file. We will run the gulp build in the next step. This is how the project tree should look at this point — files at step 0.
#Step 1: Initializing Project Files
Next, we will initialize the project files, assets folder and start writing the index.html.
touch index.html mkdir assets && touch assets/style.css mkdir src && cd src touch app.js request.js helper.js touch container.jsx filterContainer.jsx tag.jsx user.jsx userImg.jsx
We recommend taking the stylesheet file style.css and paste it as is intoassets/style.css file.
Next, we will use our project setup files and call them from index.html.
<!DOCTYPE html> <html> <head> <title>Meetup</title> <link rel="stylesheet" href="bower_components/bootstrap/dist/css/bootstrap.min.css"> <link rel="stylesheet" type="text/css" href="assets/style.css" /> <script src="bower_components/jquery/dist/jquery.min.js"></script> <script src="bower_components/appbase-js/browser/appbase.js"></script> <script src="bower_components/bootstrap/dist/js/bootstrap.min.js"></script> <script type="text/javascript" src="bower_components/typeahead.js/dist/typeahead.bundle.js"></script> <meta name="viewport" content="width=device-width, initial-scale=1"> </head> <body class="container"> <a href="https://github.com/appbaseio-apps/meetupblast-react"><img style="position: absolute; top: 0; right: 0; border: 0;z-index:15" src="https://camo.githubusercontent.com/652c5b9acfaddf3a9c326fa6bde407b87f7be0f4/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f6f72616e67655f6666373630302e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"></a> <div id="container"></div> <script type="text/javascript" src="src/helper.js"></script> <script type="text/javascript" src="src/request.js"></script> <script type="text/javascript"> var REQUEST = new meetup_request(null); </script> <script src="dist/main.js"></script> </body> </html>
Now that we have initialized all the project files, let’s transpile our non-existent .jsx files and run the app.
gulp browserify /* If you see any missing node modules like 'vinyl-source-stream', add them with an npm install command */ npm install --save vinyl-source-stream
You should now see a dist/ folder generated in the project root with a file called main.js (as expected from our gulp build process).
Let’s run the app at this point using
python -m SimpleHTTPServer 3000
Image: Step 1, a blank screen page created using our project setup
Your project tree at this point should look like this — project tree at step 1.
#Step 2: Writing UI Components with React
Our project is setup with all the necessary files, we have a good handle on the React transpiling process using gulp. Now’s the time to dive into the src/ codebase.
var React = require('react'); var ReactDOM = require('react-dom'); var Container = require('./container.jsx'); ReactDOM.render( <div> <Container></Container> </div> , document.getElementById('container'));
app.js is the entry point. In case you missed it, we used this file path in our gulp browserify build process too.
helper.js is a helper file, we recommend you get it from here and paste it as is.
It requires container.jsx file, which we declare inside the app.js file.
Container.jsx is where we define the main app container. It contains:
- filterContainer.jsx contains the city and topic filter components. It fires the RSVP feed based on current cities and topics selected.
a. tag.jsx defines the actual filter component, and the checkboxing / removing methods. - user.jsx displays the single user's RSVP feed UI.
a. userImg.jsx displays the default image if user's pic can't be found.
Before delving into the React components, here’s a small primer on them. React components are the defining units of code organization (like classes in Java) in React. A component can inherit another component, have child components and can be published on the interwebs. A component is always defined with aReact.createClass({specObject}) invocation, where the specObject should have a mandatory render() method. It can also contain other methods as well. The official docs are a good reference to whet your appetite about a component’s spec and lifecycle.
If you have never created a React component based app before, we recommend checking out this status feed app tutorial.
Enough said, let’s take a look at out codebase.
var React = require('react'); var FilterContainer = require('./filterContainer.jsx'); var User = require('./User.jsx'); var Container = React.createClass({ getInitialState: function() { return { users: [], CITY_LIST: [], TOPIC_LIST: [] }; }, componentDidMount: function() { var $this = this; this.make_responsive(); $('.meetup-record-holder').on('scroll', function() { if ($(this).scrollTop() + $(this).innerHeight() >= this.scrollHeight) { var stream_on = REQUEST.PAGINATION($this.state.CITY_LIST, $this.state.TOPIC_LIST); stream_on.done(function(res) { $this.on_get_data(res, true); }).fail('error', function(err) {}); } }); }, make_responsive: function() { function size_set() { var window_height = $(window).height() - 15; $('.meetup-record-holder').css('height', window_height); }; size_set(); $(window).resize(function() { size_set(); }); }, on_get_data: function(res, append) { var $this = this; //responseStream.stop(); if (res.hasOwnProperty('hits')) { var record_array = res.hits.hits; if (append) { var arr = $this.state.users; var new_array = $.merge(arr, record_array); $this.setState({ users: new_array }); } else { record_array = record_array.reverse(); $this.setState({ users: record_array }); } } else { var arr = $this.state.users; arr.unshift(res); $this.setState({ users: arr }); } }, set_list: function(method, list) { if(method == 'city') { this.setState({ CITY_LIST: list }); } else { this.setState({ TOPIC_LIST: list }); } }, render: function() { var $this = this; return ( <div className="row meetup-container"> <FilterContainer key='1' on_get_data={this.on_get_data} CITY_LIST={this.state.CITY_LIST} TOPIC_LIST={this.state.TOPIC_LIST} set_list={this.set_list} > </FilterContainer> <div className="meetup-record-holder" id="meetup-record-holder"> <div className="container full_row" id="record-container"> {this.state.users.map(function(single_user1, i){ var single_user = single_user1._source; return ( <User key={i} index={i} name={single_user.member.member_name} img={single_user.member.photo} event_name={single_user.event.event_name} group_city={single_user.group.group_city} group_topics={single_user.group.group_topics} event_url={single_user.event.event_url} TOPIC_LIST={$this.state.TOPIC_LIST} ></User> ); })} </div> </div> </div> ); } }); module.exports = Container;
Container is the main component. It responsible for keeping the state of three variables:
- users — an array of RSVP feeds that are being streamed by appbase.io
- CITY_LIST — an array of current city selection for filtering which feeds need to be streamed
- TOPIC_LIST — an array of current topic selection for filtering feeds by topics.
If we wanted to include other UI elements to filter feeds by (for instance, dates of meetups), we would similarly use another variable keep it’s state in the container.
The container is divided into two sub-components:
- FilterContainer component which creates the UI widget and manages the interaction flow to update the cities and topic, and
- User component is responsible for displaying the individual feed element’s UI.
var React = require('react'); var Tag = require('./tag.jsx'); var FilterContainer = React.createClass({ componentWillMount: function() { this.fire_response(); }, fire_response: function() { var $this = this; streamingClient = REQUEST.GET_STREAMING_CLIENT(); var stream_on = REQUEST.FIRE_FILTER(this.props.CITY_LIST, this.props.TOPIC_LIST); stream_on.on('data', function(res) { $this.props.on_get_data(res); $this.stream_start(); }).on('error', function(err) {}); }, stream_start: function() { var $this = this; streamingClient = REQUEST.GET_STREAMING_CLIENT(); var stream_on = REQUEST.STREAM_START(this.props.CITY_LIST, this.props.TOPIC_LIST); stream_on.on('data', function(res) { $this.props.on_get_data(res, true); }).on('error', function(err) {}); }, set_list: function(method, list) { this.props.set_list(method, list); this.fire_response(); }, render: function() { return ( <div className="meetup-filter-container"> <Tag key="0" type="city" set_list={this.set_list} list={this.props.CITY_LIST} fire_response={this.fire_response}></Tag> <Tag key="1" type="topic" set_list={this.set_list} list={this.props.TOPIC_LIST} fire_response={this.fire_response}></Tag> </div> ) } }); module.exports = FilterContainer;
In the FilterContainer, we initialize the RSVP feed stream via the fire_responsemethod. It contains the Tag component to reuse the UI elements for building the city and topic list views.
Image: City and Topic UI elements built with FilterContainer and Tag components
You can copy and paste the Tag component’s code from here.
Let’s take a look at the User component.
var React = require('react'); var UserImg = require('./userImg.jsx'); //User component var User = React.createClass({ HIGHLIGHT_TAGS: function(group_topics) { var highlight_tags = []; var group_topics = group_topics; var highlight = this.props.TOPIC_LIST; if (highlight.length) { for (i = 0; i < group_topics.length; i++) { for (var j = 0; j < highlight.length; j++) { if (highlight[j] == group_topics[i]) group_topics.splice(i, 1); } } for (i = 0; i < highlight.length; i++) { highlight_tags.push(highlight[i]); } } var lower = group_topics.length < 3 ? group_topics.length : 3; for (i = 0; i < lower; i++) { highlight_tags.push(group_topics[i]['topic_name']); } return highlight_tags; }, render: function() { var highlight_tags = this.HIGHLIGHT_TAGS(this.props.group_topics); return ( <a className="full_row single-record single_record_for_clone" href={this.props.event_url} target="_blank" key="1"> <div className="img-container"> <UserImg key={this.props.event_url} src={this.props.img} /> </div> <div className="text-container full_row"> <div className="text-head text-overflow full_row"> <span className="text-head-info text-overflow"> {this.props.name} is going to {this.props.event_name} </span> <span className="text-head-city">{this.props.group_city}</span> </div> <div className="text-description text-overflow full_row"> <ul className="highlight_tags"> { highlight_tags.map(function(tag,i){ return (<li key={i}>{tag}</li>) }) } </ul> </div> </div> </a> ) } }); module.exports = User;
A lot of the code here is in making sure we give it the proper styling layout. By now, it should be clear how components in React. They are not very different from the abstractions offered by Object Oriented Programming languages like Java.
This component uses one sub-component called UserImg. It’s a very simple component that uses a default puppy image when a user’s image URL in the RSVP JSON doesn’t resolve.
var React = require('react'); var UserImg = React.createClass({ componentDidMount: function() { var self = this; this.img = new Image(); var defaultSrc = 'http://www.avidog.com/wp-content/uploads/2015/01/BellaHead082712_11-50x65.jpg'; this.img.onerror = function() { if (self.isMounted()) { self.setState({ src: defaultSrc }); } }; this.img.src = this.state.src; }, getInitialState: function() { return { src: this.props.src }; }, render: function() { return <img src={this.state.src} />; } }); module.exports = UserImg;
This is the entirety of our React code: We should have app.js, container.jsx, filterContainer.jsx, tag.jsx, user.jsx and userImg.jsx files.
We are missing one last important file before we get to see the working demo, request.js. We use the appbase-js lib here for defining live queries on the RSVP feed data. We recommend getting it as is from here to get to the functional demo.
Let’s run the app now.
gulp browserify python -m SimpleHTTPServer 3000
Image: Step 2, complete UI rendered with React
The project tree at this step should resemble the final project. You can also see the live demo at appbaseio-apps.github.io/meetupblast-react.
How Live Queries Work
Earlier, we skipped an important part of explaining how the live queries work. Let’s take a look at it here.
There are three important queries happening in the UI:
- Streaming RSVP feeds live filtered by the user selected cities and topics.
- Generating the list of cities and topics.
- Paginating to show historical data when user scrolls down.
Let’s see how to do the first one.
// connecting to the app using our unique credentials. We will use these since the backend worker is configured to update this app, but you can change this by creating your own app from appbase.io dashboard. var appbaseRef = new Appbase({ appname: "meetup2", url: "https://qz4ZD8xq1:a0edfc7f-5611-46f6-8fe1-d4db234631f3@scalr.api.appbase.io" }) // Now let's get the initial feed data var response = appbaseRef.search({ type: "meetup", size: 25, body: { query: { match_all: {} } } }) response.on("data", function(data) { console.log("25 results matching everything", data.hits.hits) // all initial data is returned in an array called hits.hits. You can browse the data object to see other interesting meta data like the time taken for the query. });
When you run the above snippet in the console, you should see an array of size 25. Now, this is great but how do we continuously stream new RSVP feeds as they are indexed in the backend. Here’s how:
// Let's subscribe to new data updates. searchStream() works exactly like search() except it only returns new data matching the query is indexed or modified. var streamResponse = appbaseRef.searchStream({ type: "meetup", body: { query: { match_all: {} } } }) streamResponse.on("data", function(sdata) { console.log("I am a new data", sdata._source) // all new data is returned as JSON data in streams })
Image: Browser console output when running the above script
We now know how live queries work with appbase.io APIs. The interesting part here is the JSON query object defined in the search() and searchStream() methods. We can change this query object to only showing RSVPs that match a white list of cities and/or topics. Anything that can be specified with Elasticsearch’s Query DSL can be queried live with appbase.io. This is exactly what request.js does. It also creates a list of top 1000 cities and topics at the start of the app based on aggregating all the RSVPs indexed in the last month. See https://github.com/appbaseio-apps/meetupblast-react/blob/master/src/request.js#L17. You can read more about appbase’s APIs here or check out a whole host of interesting live querying based apps at appbase.io’s community page.
#Summary
This is Part-II of a N-Part series where we show how to build a Twitter like Live Search Feed using appbase.io and React.JS.
- We started out with understanding the feed datastructure,
- We then looked at the backend worker, where we fetched data from meetup’s streaming endpoint and indexed it into appbase.io,
- Finally, we looked at building the live feed user interface using React Components. We covered the gulp build system for transpiling React instep 0, linked everything together in step 1 and wrote React Components for the UI in step 2.
Live demo and final code repository links for further hacking.