Beginner’s Guide to understand Kafka

It’s a digital age. Wherever there is data, we hear about Kafka these days. One of my projects I work, involves entire data system (Java backend) that leverages Kafka to achieve what deals with tonnes of data through various channels and departments. While working on it, I thought of exploring the setup in Windows. Thus, this guide helps learn Kafka and showcases the setup and test of data pipeline in Windows.

Introduction

<kafka-logo>
An OpenSource Project in Java & Scala

Apache Kafka is a distributed streaming platform with three key capabilities:

  • Messaging system – Publish-Subscribe to stream of records
  • Availability & Reliability – Store streams of records in a fault tolerant durable way
  • Scalable & Real time – Process streams of records as they occur

Data system components

Kafka is generally used to stream data into applications, data lakes and real-time stream analytics systems.

<kafka-highlevel-architecture>

Application inputs messages onto the Kafka server. These messages can be any defined information planned to capture. It is passed across in a reliable (due to distributed Kafka architecture) way to another application or service to process or re-process them.

Internally, Kafka uses a data structure to manage its messages. These messages have a retention policy applied at a unit level of this data structure. Retention is configurable – time based or size based. By default, the data sent is stored for 168 hours (7 days).

Kafka Architecture

Typically, there would be multiples of producers, consumers, clusters working with messages across. Horizontal scaling can be easily done by adding more brokers. Diagram below depicts the sample architecture:

kafka-internals

Kafka communicates between the clients and servers with TCP protocol. For more details, refer: Kafka Protocol Guide

Kafka ecosystem provides REST proxy that allows an easy integration via HTTP and JSON too.

Primarily it has four key APIs: Producer API, Consumer API, Streams API, Connector API

Key Components & related terminology
  • Messages/Records – byte arrays of an object. Consists of a key, value & timestamp
  • Topic – feeds of messages in categories
  • Producer – processes that publish messages to a Kafka topic
  • Consumer – processes that subscribe to topics and process the feed of published messages
  • Broker – It hosts topics. Also referred as Kafka Server or Kafka Node
  • Cluster – comprises one or more brokers
  • Zookeeper – keeps the state of the cluster (brokers, topics, consumers)
  • Connector – connect topics to existing applications or data systems
  • Stream Processor – consumes an input stream from a topic and produces an output stream to an output topic
  • ISR (In-Sync Replica) – replication to support failover.
  • Controller – broker in a cluster responsible for maintaining the leader/follower relationship for all the partitions
Zookeeper

Apache ZooKeeper is an open source that helps build distributed applications. It’s a centralized service for maintaining configuration information. It holds responsibilities like:

  • Broker state – maintains list of active brokers and which cluster they are part of
  • Topics configured – maintains list of all topics, number of partitions for each topic, location of all replicas, who is the preferred leader, list of ISR for partitions
  • Controller election – selects a new controller whenever a node shuts down. Also, makes sure that there is only one controller at any given time
  • ACL info – maintains Access control lists (ACLs) for all the topics

Kafka Internals

Brokers in a cluster are differentiated based on an ID which typically are unique numbers. Connecting to one broker bootstraps a client to the entire Kafka cluster. They receive messages from producers and allow consumers to fetch messages by topic, partition and offset.

A Topic is spread across a Kafka cluster as a logical group of one or more partitions. A partition is defined as an ordered sequence of messages that are distributed across multiple brokers. The number of partitions per topic are configurable during creation.

Producers write to Topics. Consumers read from Topics.

<kafka-partition>

Kafka uses Log data structure to manage its messages. Log data structure is an ordered set of Segments that are collection of messages. Each segment has files that help locate a message:

  1. Log file – stores message
  2. Index file – stores message offset and its starting position in the log file

Kafka appends records from a producer to the end of a topic log. Consumers can read from any committed offset and are allowed to read from any offset point they choose. The record is considered committed only when all ISRs for partition write to their log.

leader-follower

Among the multiple partitions, there is one leader and remaining are replicas/followers to serve as back up. If a leader fails, an ISR is chosen as a new leader. Leader performs all reads and writes to a particular topic partition. Followers passively replicate the leader. Consumers are allowed to read only from the leader partition.

A leader and follower of a partition can never reside on the same node.

leader-follower2

Kafka also supports log compaction for records. With it, Kafka will keep the latest version of a record and delete the older versions. This leads to a granular retention mechanism where the last update for each key is kept.

Offset manager is responsible for storing, fetching and maintaining consumer offsets. Every live broker has one instance of an offset manager. By default, consumer is configured to use an automatic commit policy of periodic interval. Alternatively, consumer can use a commit API for manual offset management.

Kafka uses a particular topic, __consumer_offsets, to save consumer offsets. This offset records the read location of each consumer in each group. This helps a consumer to trace back its last location in case of need. With committing offsets to the broker, consumer no longer depends on ZooKeeper.

Older versions of Kafka (pre 0.9) stored offsets in ZooKeeper only, while newer version of Kafka, by default stores offsets in an internal Kafka topic __consumer_offsets

consumer-groups

Kafka allows consumer groups to read data in parallel from a topic. All the consumers in a group has same group ID. At a time, only one consumer from a group can consume messages from a partition to guarantee the order of reading messages from a partition. A consumer can read from more than one partition.

Kafka Setup On Windows

setup-on-windows
Pre-Requisite
Setup files
  1. Install JRE – default settings should be fine
  2. Un-tar Kafka files at C:\Installs (could be any location by choice). All the required script files for Kafka data pipeline setup will be located at: C:\Installs\kafka_2.12-2.5.0\bin\windows
  3. Configuration changes as per Windows need
    • Setup for Kafka logs – Create a folder ‘logs’ at location C:\Installs\kafka_2.12-2.5.0
    • Set this logs folder location in Kafka config file: C:\Installs\kafka_2.12-2.5.0\config\server.properties as log.dirs=C:\Installs\kafka_2.12-2.5.0\logs
    • Setup for Zookeeper data – Create a folder ‘data’ at location C:\Installs\kafka_2.12-2.5.0
    • Set this data folder location in Zookeeper config file: C:\Installs\kafka_2.12-2.5.0\config\zookeeper.properties as dataDir=C:\Installs\kafka_2.12-2.5.0\data
Execute
  1. ZooKeeper – Get a quick-and-dirty single-node ZooKeeper instance using the convenience script already packaged along with Kafka files.
    • Open a command prompt and move to location: C:\Installs\kafka_2.12-2.5.0\bin\windows
    • Execute script: zookeeper-server-start.bat C:\Installs\kafka_2.12-2.5.0\config\zookeeper.properties
    • ZooKeeper started at localhost:2181. Keep it running.
      demo-zookeeper
  2. Kafka Server – Get a single-node Kafka instance.
    • Open another command prompt and move to location: C:\Installs\kafka_2.12-2.5.0\bin\windows
    • ZooKeeper is already configured in the properties file as zookeeper.connect=localhost:2181
    • Execute script: kafka-server-start.bat C:\Installs\kafka_2.12-2.5.0\config\server.properties
    • Kafka server started at localhost: 9092. Keep it running.
      demo-kafka
      Now, topics can be created and messages can be stored. We can produce and consume data from any client. We will use command prompt for now.
  3. Topic – Create a topic named ‘testkafka’
    • Use replication factor as 1 & partitions as 1 given we have made a single instance node
    • Open another command prompt and move to location: C:\Installs\kafka_2.12-2.5.0\bin\windows
    • Execute script: kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic testkafka
    • Execute script to see created topic: kafka-topics.bat --list --bootstrap-server localhost:9092
      demo-topic
    • Keep the command prompt open just in case.
  4. Producer – setup to send messages to the server
    • Open another command prompt and move to location: C:\Installs\kafka_2.12-2.5.0\bin\windows
    • Execute script: kafka-console-producer.bat --bootstrap-server localhost:9092 --topic testkafka
    • It will show a ‘>’ as a prompt to type a message. Type: “Kafka demo – Message from server”
      demo-producer
    • Keep the command prompt open. We will come back to it to push more messages
  5. Consumer – setup to receive messages from the server
    • Open another command prompt and move to location: C:\Installs\kafka_2.12-2.5.0\bin\windows
    • Execute script: kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic testkafka --from-beginning
    • You would see the Producer sent message in this command prompt window – “Kafka demo – Message from server”
      demo-consumer
    • Go back to Producer command prompt and type any other message to see them appearing real time in Consumer command prompt
      kafka-demo
  6. Check/Observe – few key changes behind the scene
    • Files under topic created – they keep track of the messages pushed for a given topic
      topic-files
    • Data inside the log file – All the messages that are pushed by producer are stored here
      topic-log
    • Topics present in Kafka – once a consumer starts reading messages from topic, __consumer_offsets is automatically created as a topic
      topic-present

NOTE: In case you want to choose Zookeeper to store topics instead of Kafka server, it would require following script commands:

  • Topic create: kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testkafka
  • Topics view: kafka-topics.bat --list --zookeeper localhost:2181

With above, we are able to see messages sent by Producer and received by Consumer using a Kafka setup.

When I tried to setup Kafka, I faced few issues on the way. I have documented them for reference to learn. This should also help others if they face something similar: Troubleshoot: Kafka setup on Windows.

One should not encounter any issues with below shared files and the steps/commands shared above.

Download entire modified setup files for Windows from here: https://github.com/sandeep-mewara/kafka-demo-windows

hurray

References:
https://kafka.apache.org
https://cwiki.apache.org/confluence/display/KAFKA
https://docs.confluent.io/2.0.0/clients/consumer.html

Make browser a basic html editor

While working on one of my recent blogs, I stumbled upon an HTML DOM property that looked interesting.

In past,

  • I have to see how the text change looks in a webpage – make change, refresh page or run the application again
  • I have to inspect, find the related DOM to make any text change to it and then write code to make the change to see it
  • I downloaded HTML page and then made some change in its text to add/edit/remove some comments for clean print.
  • Have some logic to provide an editable HTML page to users

Well, no more. Seems we have a new property (surely it was not there few years back but introduced recently): document.designMode

I tried in Firefox, from menu items, go to: Tools -> Web Developer -> Web Console. Write:

document.designMode = "on"

Post this, you can edit the webpage text right in your browser!

documentModeEx

Sample real use case could be providing a portion of page editable to users. Add that in an iframe and then turn the designMode of that to ‘on’:

iframeNode.contentDocument.designMode = "on";

A string indicating whether designMode is (or should be) set to on or off. Valid values are on and off

In IE, it would be under Developer Tools, and so on for other browsers.

design-mode
Browser Compatibility

Nice to have something like it to to convert browser into a basic HTML editor! Keep learning.

Reference: https://developer.mozilla.org/en-US/docs/Web/API/Document/designMode

Quick look into SignalR

Last year, I was looking into various ways of communication between client and server for one of my project work. Evaluated SignalR too as one of the options. I found decent documentation online to put across a test app to see how it works.

Introduction

SignalR is a free and open-source software library for Microsoft ASP.NET that provides ability to server-side code push content to the connected clients in real-time.

Pushing data from the server to the client (not just browser clients) has always been a tough problem. SignalR makes it dead easy and handles all the heavy lifting for you.

https://github.com/SignalR/SignalR

Detailed documentation can be found at: https://dotnet.microsoft.com/apps/aspnet/signalr

Most of the shareout online considers it as ASP.NET/Web application solution only which is not true. As mentioned in quote above, client could be a non-browser too like a desktop application.

I gathered that behind the scenes, SignalR primarily tries to use Web Socket protocol to send data. WebSocket is a new HTML5 API (refer my detailed article on it) that enables bi-directional communication between the client and server. In case of any compatibility gap, SignalR uses other protocols like long polling.

P.S.: Since most of the code to make the test app was picked from web, all the credit to them. One specific source that I can recall: https://docs.microsoft.com/en-us/aspnet/signalr/overview/deployment/tutorial-signalr-self-host

Now, a quick peek into the SignalR test app
  • Using the SignalR Hub class to setup the server. Defined a hub that exposes a method like an end point for clients to send a message. Server can process the message and send back a response to all or few of the clients.
[HubName("TestHub")]
public class TestHub : Hub
{
    public void ProcessMessageFromClient(string message)
    {
        Console.WriteLine($"<Client sent:> {message}");

        // Do some processing with the client request and send the response back
        string newMessage = $"<Service sent>: Client message back in upper case: {message.ToUpper()}";
        Clients.All.ResponseToClientMessage(newMessage);
    }
}

  • Specify which “origins” we want to allow our application to accept. It is setup via CORS configuration. CORS is a security concept that allows end points from different domains to interact with each other.
public void Configuration(IAppBuilder app)
{
     app.UseCors(CorsOptions.AllowAll);
     app.MapSignalR();
}

  • Server is setup using OWIN (Open Web Interface for .NET). OWIN defines an abstraction between .NET web servers and web applications. This helps in self-hosting a web application in a process, outside of IIS.
static void Main(string[] args)
{
    string url = @"http://localhost:8080/";

    using (WebApp.Start<Startup>(url))
    {
        Console.WriteLine($"============ SERVER ============");
        Console.WriteLine($"Server running at {url}");
        Console.WriteLine("Wait for clients message requests for server to respond OR");
        Console.WriteLine("Type any message - it will be broadcast to all clients.");
        Console.WriteLine($"================================");

        // For server broadcast test
        // Get hub context 
        IHubContext ctx = GlobalHost.ConnectionManager.GetHubContext<TestHub>();

        string line = null;
        while ((line = Console.ReadLine()) != null)
        {
            string newMessage = $"<Server sent:> {line}";
            ctx.Clients.All.MessageFromServer(newMessage);
        }

        // pause to allow clients to receive
        Console.ReadLine();
    }
}

In above code, Using IHubContext:

  1. Server is setup to have earlier defined Hub as one of the broadcast end point.
  2. Server is also setup to broadcast any message to all clients by itself if needed be

  • Client is setup to communicate with the server (send and receive message) via hub using HubConnection & IHubProxy. Client can invoke the exposed end point in the hub to a send a message.
static void Main(string[] args)
{
    string url = @"http://localhost:8080/";

    var connection = new HubConnection(url);
    IHubProxy _hub = connection.CreateHubProxy("TestHub");
    connection.Start().Wait();

    // For server side initiation of messages
    _hub.On("MessageFromServer", x => Console.WriteLine(x));
    _hub.On("ResponseToClientMessage", x => Console.WriteLine(x));

    Console.WriteLine($"============ CLIENT ============");
    Console.WriteLine("Type any message - it will be sent as a request to server for a response.");
    Console.WriteLine($"================================");

    string line = null;
    while ((line = Console.ReadLine()) != null)
    {
        // Send message to Server
        _hub.Invoke("ProcessMessageFromClient", line).Wait();
    }

    Console.Read();
}

With above setup, we can see the communication between Client & Server realtime like below:

SignalR test

Things to consider while opting for SignalR

SignalR looks awesome. But, there are couple of things one should know while using it:

  • It’s a connected technology – each client connected through SignalR is using a persistent and dedicated connection on the Web Server. SignalR is shared as scalable but it never harms to look for any queries around connections, memory, cpu, etc with higher number of clients.
  • One would need to setup a valid host server with a PORT open on it that can be used for communication. This could depend on an organisation security protocols and approvals.

Hope this gives a quick overview about SignalR. Keep learning!

Download entire code for lookup from here: https://github.com/sandeep-mewara/SignalRTest

Beginner’s quick start to learn React.js

I recently experimented with React.js, so thought of sharing key points that I learnt. Though there are handful of material online, couldn’t find one that covers all in a concise way that can help learn key aspects of ReactJS. I believe this would resonate with few and help them learn, understand and get a jumpstart with ReactJS.

react-js
What is React.js?

React.js is an open source JavaScript based library for building frontend (user interface) of a web or mobile application.

Why React.js?

Every web application core is to have a fast rendering response for better user experience. Because of this ease, users come back often and it leads higher usage and adaptability.

Further, based on how it achieves speed, it is scalable and reusable.

How React.js does it?

React.js works at component level. It helps break an app into many small components with their own responsibilities. This makes things simpler and scalable. With this breakdown,

  • it’s easier to refresh/update a portion of view without reloading an entire page of the app.
  • it leads to build once and reuse across.

Another key part of React.js is being declarative. There is a an abstraction from details on how to do. This makes it easier to read and understand.

A declarative example would be telling my son to make a house craft from paper instead of guiding him with each step of how to get the paper, cut it, paste it to form a house craft. Of course the assumption here has to be true that my son knows how to make it.

A quick comparison with jQuery here (it’s imperative) – it would need details on how to build the house craft.

Translating above in Javascript langauge world:

  • With React – we define on how we want a particular component to be rendered and we never interact with DOM to reference later
  • With jQuery – we would tell browser exactly what needs to be done using DOM elements or events need basis
Key features

Following features help us achieve above:

  • Components – Simple or State

These are small reusable codes that returns a React element to render. This component can have state related aspect based on need.

// Simple component - a Function Component
// props - input to React component - data passed from parent caller
function ComponentExample(props) {
  return <h1>Hola! {props.name}</h1>;
}

// Simple component - a Class Component
class ComponentExample extends React.Component {
  render() {
    return <h2>Hola! {this.props.name}</h2>;
  }
}

// State based component
// Needed when data associated with component change over time
class ComponentExample extends React.Component {
  constructor(props) {
    super(props);
    this.state = {author: "Sandeep Mewara"};
  }
  render() {
    return (
      <div>
        <h2>Hola! {this.props.name}</h2>
        <p>Author: {this.state.author}</p>
      </div>
   );
  }
}

For above example component, use normal HTML syntax: <ComponentExample />

  • Virtual DOM

DOM (Document Object Model) is a structured representation of the HTML elements present on a web page. Traditionally, one would need to get elements out of DOM to make any change. In context of an area of a webpage, it would need a lot more work to refresh it with updated content when needed.

React helps here with its declarative API. A copy of actual DOM is kept in memory which is much faster to change. Once done, React uses its ReactDOM library to sync the virtual representation of UI in memory to the actual DOM.

ReactDOM library internally keeps two VDOMs – one before update and one after. With them, React knows exactly what all to be updated in actual DOM and does all of it on the fly leading much faster updates compared to traditional DOM updates.

React.js has a library ReactDOM to access and modify the actual DOM.

To render HTML on a webpage, use: ReactDOM.render()

  • JSX (JavaScript eXtension)

JSX is a syntax extension to JavaScript that follows XML rules. It’s more of a helpful tool than requirement in React as mentioned below in their website:

React doesn’t require using JSX, but most people find it helpful as a visual aid when working with UI inside the JavaScript code

JSX converts HTML tags into React elements that are placed in DOM without any commands like createElements(), etc.

// Example with JSX
const testHtml = <h2>Hola! Sandeep Mewara</h2>;
ReactDOM.render(testHtml, document.getElementById('root'));

// Same above example without JSX
const testHtml = React.createElement('h2', {}, 'Hola! Sandeep Mewara');
ReactDOM.render(testHtml, document.getElementById('root'));

Normally, we can’t assign an HTML tag to a JavaScript variable but we can with JSX!

  • Unidirectional data flow

React implements one way reactive data flow. It uses flux as a pattern to keep data unidirectional. Interpret it as you often nest child components within higher order parent components. Snapshot of state is passed across from parent to child components via props (readonly, cannot be updated) and updates from child to parent happen via callbacks bound to some control on child component.

  • ES6 compatible

React library is ES6 (ECMAScript 2015 or JavaScript 6) enabled and thus makes it easier to write code in React. Among all changes to standardize JavaScript in ES6Classes introduction is one of them which plays a critical role in React.

  • Lifecycle

Each React component has a lifecycle that helps write a code at a specific time during the flow as per need.

// Use class for any local state & lifecycle hooks
class TestClass extends Component 
{  
    // first call to component when it is initiated
    constructor(props) 
    {    
        // with it, 'this' would refer to this component
        super(props); 
        // some local state initialized 
        this.state = {currentdate: new Date()};
    };   
    
    // executes before the initial render
    componentWillMount() {    
     
    };  
    
    // executes after the initial render
    componentDidMount() {  

    };

    // executes when component gets new props
    componentWillReceiveProps() {   
          
    };

    // executes before rendering with new props or state
    shouldComponentUpdate() {   
          
    };
    
    // executes before rendering with new props or state
    componentWillUpdate() {   
        
    };

    // executes after rendering with new props or state
    componentDidUpdate() {   
          
    };
    
    // executes before component is removed from DOM
    componentWillUnmount() {   
          
    };

    // HTML to be displayed by the component rendering 
    render() 
    {    
        return (      
            <h1>Current Date: {this.state.currentdate.toString()}</h1>
        );  
    }; 
}

For entire React glossary, please refer: https://reactjs.org/docs/glossary.html

Sample application Setup

We will explore and understand more from React’s demo app. We will jump start our sample app bootstrapped with Create React App

I used yarn create react-app demo-react-app and opened the created directory in IDE that looked like:

default react project structure

With above, once I ran yarn start in root folder demo-react-app, app was up and running without any code change. We can see default app hosted in browser at following url: http://localhost:3000/

default home page

Quick look at few key files that connect dots that lead to above UI view:

  • public/index.html

Base file which we browse using url. We see the HTML defined in it. For now, the element to notice would be a div named root.

  • src/index.js

Located at root of app, is like an entry file (like main) for app that has code like below:

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import App from './App';
import * as serviceWorker from './serviceWorker';

ReactDOM.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>,
  document.getElementById('root')
);

// If you want your app to work offline and load faster, you can change
// unregister() to register() below. Note this comes with some pitfalls.
// Learn more about service workers: https://bit.ly/CRA-PWA
serviceWorker.unregister();

It imported React and related library, CSS file for app, a component named App. After this, it defines a render method which displays whatever is defined in component App as page root element.

  • src/App.js

Defines a function component of React that returns an html with React logo and a link to render.

import React from 'react';
import logo from './logo.svg';
import './App.css';

function App() {
  return (
    <div className="App">
      <header className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <p>
          Edit <code>src/App.js</code> and save to reload.
        </p>
        <a
          className="App-link"
          href="https://reactjs.org"
          target="_blank"
          rel="noopener noreferrer"
        >
          Learn React
        </a>
      </header>
    </div>
  );
}

export default App; 

How did index.js got connected with index.html?

Create React App uses Webpack with html-webpack-plugin underneath. This Webpack uses src/index.js as an entry point. Because of this, index.js comes into picture and all other modules referenced in it. With html-webpack-plugin configuration, it automatically adds the script tag in html page.

Let’s see with few modifications to the app now!

Specifically I will be changing flavour of above 3 files to play around.

  1. AppHola.js file for a HelloWorld kind of change – displays my name instead of other texts
  2. AppNavigation.js (has portion of pages updated)
    • Introduction – simple display of texts
    • Clock/counters auto updating
    • Random color generator that updates background color of defined area
demo-app-gif

Given this was for beginners, I have not added too much of complexity to the app. I have tried to keep it as simple possible with some variance of what all can be tried.

There are plenty of imports that can be used. For example, in our demo app, to have navigation, we have used a navigation router react-router-dom import (run npm i react-router-dom --save inside root folder).

Hope this short guide/tutorial gives a broad overview about React.JS and how to start development of the same. Keep learning!

Download entire code for lookup from here: https://github.com/sandeep-mewara/demo-react-app

HttpClient single instance or multiple

Recently, we were working on a project that needed numerous HTTP requests to be made. Initial implementation had a new HttpClient object being created for every request being made. It looked to have some performance cost attached to it that led us to evaluate the effect of using single vs multiple instances of HttpClient.

Problem Statement:

Whats the best way to use HttpClient for multiple requests and the performance cost associated with it?

Assessment:

Went through the Microsoft documentation, which seemed updated based on last when I read few years back. Found a fineprint for myself that states:

HttpClient is intended to be instantiated once and re-used throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. This will result in SocketException errors.

This was a straight give away that we should use a single instance HttpClient – irrespective of a usecase, one would want to keep distance from SocketException errors (though probability of it would be high for heavy usage of HTTP requests).

Now, the query was how to have single HttpClient for multiple requests but with different request payload for the calls? Also, does this has any impact on performance of the calls and if so, how much?

Resolution:

I started with looking into performance aspect for the two options. Created a test application that helped evaluate the time taken for various number of requests. Tried with www.google.com but seems they have some kind of check at 1000 requests so went ahead with www.bing.com that looked uniform till 5000 requests that I tried with.

for (var i = 0; i < noOfConnections; i++)
{
    using (var httpClient = new HttpClient())
    {
        var result = httpClient.GetAsync(new Uri("http://www.bing.com/")).Result;
    }
}
//having private static readonly HttpClient _httpClient = new HttpClient();
for (var i = 0; i < noOfConnections; i++)
{
    var result = _httpClient.GetAsync(new Uri("http://www.bing.com/")).Result;
}

With the above, I got the following numbers on an average post few runs:

No of RequestsMultiple Instance (s)Single Instance (s)%age Diff
1002016.6716.65
5001038814.56
100021617419.44
200043035118.37
5000103290612.21

It looked like the difference peaked around 1000 requests and overall there was an improvement with single instance.

Now, given we had a usecase where multiple HTTP requests has to be made simultaneously but with different payloads, looked at how to achieve it with single instance. Keeping multiple types of requests, unit testing, high load – One possible way looked like below that worked out well for us:

// Single instance of HttpClientManager was setup
public class HttpClientManager : IHttpClientManager
{
    ...
    public HttpClientManager(HttpMessageHandler messageHandler)
    {
        _httpClient = new HttpClient(messageHandler);
    }
    private HttpRequestMessage SetupRequest(IRequestPayload requestPayload)
    {
        var request = new HttpRequestMessage
        {
            RequestUri = new Uri(requestPayload.Url)
        };
        switch (requestPayload.RequestType)
        {
            case RequestType.POST_ASYNC:
                request.Method = HttpMethod.Post;
                request.Content = GetHttpContent(requestPayload.ContentJson);
                break;
            case RequestType.PUT_ASYNC:
                request.Method = HttpMethod.Put;
                request.Content = GetHttpContent(requestPayload.ContentJson);
                break;
            case RequestType.DELETE_ASYNC:
                request.Method = HttpMethod.Delete;
                break;
            case RequestType.GET_ASYNC:
                request.Method = HttpMethod.Get;
                break;
            default:
                request.Method = HttpMethod.Get;
                break;
        }
        ...
    }
    public HttpResponseMessage ExecuteRequest(IRequestPayload requestPayload)
    {
        HttpRequestMessage httpRequestMessage = SetupRequest(requestPayload);
        HttpResponseMessage httpResponseMessage = _httpClient.SendAsync(httpRequestMessage, HttpCompletionOption.ResponseHeadersRead).Result;
        return httpResponseMessage;
    }
    private HttpContent GetHttpContent(string contentJson)
    {
        return new StringContent(contentJson, ENCODING, MEDIATYPE_JSON);
    }
}

Since there are numerous articles on the web explaining details of the entire HttpClient workflow and inner details, I will not cover that here but a quick explanation on couple of key info. In the code above:

HttpRequestMessage is used to setup HttpClient object based on our need. We make use of the fact that HttpRequestMessage can be used only once. After the request is sent, it is disposed immediately to ensure that any associated Content object is disposed.

Making use of HttpClient underlying implementation, have used HttpMessageHandler more from the unit test point of view.

Conclusion:

One should use a single instance of HttpClient at application level to avoid create/destroy of it multiple times. Further, results suggest this also has better performance with more than 12% improvement based on the load.

For multiple requests of different payloads, having a single instance HttpClient but a new HttpRequestMessage for every request looked a good approach to use.

P.S.: For .NET Core, Microsoft added a new interface around the same discussion to have better handle at HttpClient instance: https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests

Entire code for lookup can be downloaded from here.

HTML5: Tips !!!

Long time, since I have blogged. Specifically, this particular post – had all the details and wanted to post more than a year back. I guess, better late than never!
Here are few of the HTML5 tips that might help while using it’s various features:

HTML5 WebSocket Protocol is present only from IIS 8.0

HTML5 Web Socket provides a two-way communication over one TCP connection (socket). IIS 8.0 (ships with Windows 8) is capable of accepting Web Socket communications. In order to use the new protocol, one has to enable it in IIS 8.0, using option “Turn Windows features on or off” present in the control panel. It should look like below:
WebSocketIIS8
A quick detail on how WebSocket works can be looked here in my article: http://www.codeproject.com/Articles/546960/HTML-Quick-Start-Web-Application#websocket
 

One can support new HTML5 elements in older browsers using HTML5 shiv

Older browsers that are still widely used by end users (namely IE6/7/8) do not have support for the new elements in HTML5. One great way to enable HTML5 element support in IE6/7/8 is to have the http://code.google.com/p/html5shiv/
As noted on the linked Google page, “shiv” and “shim” are interchangeable terms in this context.
In case interested, I came across a good sample example and details out here: http://www.sitepoint.com/html5-older-browsers-and-the-shiv/
 

Elements in HTML5 can easily be made draggable

The specifications defines an event-based mechanism – a JavaScript API and additional markup for declaring it. Any type of element can be marked draggable on a page. It’s a known fact that having a native browser support for a particular feature is best as they would be faster and provide a more responsive application.
A quick detail about the feature can be read here: http://www.codeproject.com/Articles/546960/HTML-Quick-Start-Web-Application#dragdrop
 

‘Type’ attribute for script and link are no more required in HTML5

Type attribute is optional and one does not need to provide it for script and link tag in HTML5.
Earlier, one used to give:

<link rel="stylesheet" href="somepath/somestylesheet.css" type="text/css" />
<script type="text/javascript" src="somepath/somescript.js" />

Now, in HTML5, same references can be provided as:

<link rel="stylesheet" href="somepath/somestylesheet.css" />
<script src="somepath/somescript.js" />

This makes things simpler as the file extension and the tag were enough to interpret the type.
 

async attribute has been added to <script> tag in HTML5

HTML5 has added a new attribute for <script> tag, which is async. With this attribute, the external script will be executed asynchronously as soon as it is available. It has no effect on inline scripts.
Inline scripts and scripts without async or defer attributes are fetched and executed immediately, before the browser continues to parse the page.
 

In HMTL5, all unknown element in all browsers are displayed inline by default

The three most commonly used values are none, block, and inline. All elements have an initial or default state for their display value.For all HTML5 elements, by default they have a computed display value of inline.
 

In HTML5, page and worker do not share the same instance

In HTML5, page and worker do not share the same instance – a duplicate is created on each pass. Data passed between the main page and workers are copied. Objects are serialized as they’re handed to the worker, and subsequently, deserialized on the other end. I read that most browsers implement this feature as structured cloning. (The structured clone algorithm is a new algorithm defined by the HTML5 specification for serializing complex JavaScript objects.)
A quick detail about the feature can be read here: http://www.codeproject.com/Articles/546960/HTML-Quick-Start-Web-Application#webworkers
 

In HMTL5 Offline, relative URLs listed in a manifest are resolved relative to the manifest file

Yes, the urls in the manifest are relative to manifest’s own url. It is not to be confused that they are relative to root. Though it might happen that the urls relative to root work good because the manifest file might be located in the application root.
A quick detail about the feature can be read here: http://www.codeproject.com/Articles/546960/HTML-Quick-Start-Web-Application#offlineapp

HTML5 Quick Start Web Application

HTML5 Quick Start


I was learning HTML5 features since December. While going through it, I was playing around making a sample web application fully HTML5 enabled – kind of self learning kit that would give a basic knowledge to anyone who goes through it. Plan was to publish it as an article so that others can learn quickly and have basic features at one place. One can play around with the feature implementation straight away post download.

It took me some time to write the article as it covered good number of features. I finished working on it and have published it today on CodeProject: HTML5 Quick Start Web Application

One can have a look at it there and provide feedback.

UPDATE: Missed uploading this to Github earlier, uploaded here: https://github.com/sandeep-mewara/HTML5QuickStart

Keep learning!