Agentic AI for Beginners: My Journey into Building with Claude

There’s been a lot of buzz around Agentic AI lately, especially around how powerful Claude can be when used beyond simple prompting. Naturally, I got curious.

agentic-ai-claude.jpeg


As an architect, I wanted to understand what “agentic” really means in practice. What changes when we move from prompts to agents? And what does that mean for how we design systems? As I started exploring, it became clear this isn’t just about smarter chatbots, it’s something more.

From Prompts to Agents: What’s the Difference?

Before diving in, let’s distinguish between Generative AI and Agentic AI.

  • Generative AI (Reactive) – Deals with Prompts, where we provide an input and the model provides a one-time response. We are the orchestrator.


  • Agentic AI (Proactive) – Deals with Agents, where we provide a goal and the model determines the steps, uses tools and iterates until the goal is met. The model is the orchestrator.

“Agentic” means moving from chatting to delegating. It’s the difference between asking for instructions and having the task completed for you, like getting a recipe vs hiring a chef or asking for directions vs being driven there.

The “Agentic” Starter Pack

I started with the basics to see how Claude handles the “plumbing” of a real-world project. My exploration focused on three core hyped items:

  • Agentic Implementation: I moved away from “one-off” prompts and built loops where Claude runs a Plan -> Execute -> Test -> Fix cycle autonomously.


  • Model Context Protocol (MCP): I hooked Claude up to my local filesystem, Slack & GitHub. This was to see how the agent “reaches out” and queries the data it needs directly.


  • Role-Based Division: I experimented with “Agent Teams” by giving different Claude instances specific roles: one as the Architect to handle planning and another as the Developer to handle implementation. Further, tried to put multiple hats for the clarity of work distribution and decision making for the agent.

My Learning Project: Endpoint Watch Agent (EWA)

The goal of this project was to build a hands-on learning kit for agentic systems. Endpoint Watch Agent (EWA) is a Python-based agent that continuously monitors configurable endpoints (websites or APIs). When an endpoint is down or unhealthy, the agent autonomously evaluates the incident, avoids duplicate alerts, creates a ticket and sends a contextual Slack notification.

Flow Diagram Plan

ewa-flow-diagram.png


Structuring the Workflow

Starting from nothing, I worked with Claude itself to set up the structure and segregation of components, defining single responsibility. To keep things simple, created a single agent (Orchestrator) that runs one sequential loop: Check Endpoint 1 → Decide → Act → Check Endpoint 2 → Decide → Act...

The PolicyEngine is not an agent but a pure function called by the agent. The tools are interfaces that the agent dispatches, while the MCP servers are external services.

Explore or build on the Project available here: [Github Link]

What I Learnt: The “Pro” Framework

The real breakthrough wasn’t the model itself, but how I structured the project to guide it. Below project structure can be considered a good architectural template as a baseline start for any agentic development. The architectural pattern supports a clean separation of concerns, where we can add new tools, policy rules or tests without needing to restructure the entire system.


As a production-ready baseline though, it has gaps: no tests, single-threaded endpoint checking, no metrics, no graceful shutdown. These are solvable without rethinking the architecture, but they’d need to be added before shipping anything real.

I found that following four pillars are essential for any agentic workflow:

CLAUDE.md (The Project Brain)

This file lives in the root of your repo as the AI’s operating manual. It tells Claude agent who it is and how it should behave in this specific codebase. Thus, it helps to start with shared context instead of inferring everything from scratch each session.

# Project Context: Endpoint Watch Agent (EWA)

## Role & Mission
You are the **EWA Specialist**. Your goal is to maintain a high-availability monitoring system. You prioritize accuracy in incident detection and clarity in Slack notifications.

## Tech Stack
- **Runtime:** Python 3.12
- **Logic:** Policy-based reasoning (PolicyEngine)
- **Integrations:** Slack (Alerts), Jira (Tickets), GitHub (MCP)

## Architecture Rules
- **Separation of Concerns:** Keep tools in '/tools', logic in '/engine'.
- **Async First:** Use 'asyncio' for all network-bound endpoint checks.
- **No Deletions:** Never delete incident logs, only archive or update status.

## Dev Commands
- **Run:** 'python main.py'

CLAUDE.md is the interface between the human who designed the system and the AI that extends it. It’s not a documentation for users of the tool instead is a documentation for the next builder, human or AI.

SKILLS.md (The Capability Manual)

While CLAUDE.md is about the project, SKILLS.md is about what the agent is capable of doing. It provides pre-verified “recipes” for complex tasks, stopping the agent from hallucinating its own (often broken) logic.

# Agent Skills

## Skill: Incident Evaluation
- **When:** An endpoint returns a non-200 status.
- **Action:** 
  1. Check 'storage/incidents.json' for active tickets.
  2. If new, invoke the 'JiraTool' to create a "Critical" task.

## Skill: Slack Formatting
- **Constraint:** Always include the Status Code, Response Time, and the "Runbook Link" from the configuration file.
- **Tone:** Professional and urgent.

These are the procedural instructions or documentation that teach the agent how to use a tool effectively in a specific context.

“Plan, then Execute” Workflow

I stopped asking Claude to “just do it”. Instead, I enforced a mandatory two-step gate:

  1. The Plan: Claude must output a step-by-step technical plan first.
  2. The Approval: I review the plan for architectural alignment.
  3. The Execution: Only after approval does the agent start writing code. This eliminates 90% of the “rabbit holes” agents often fall into.

Verification Criteria

Never ask an agent to “fix a bug”. Instead, ask it to “Fix the bug and provide the specific CLI command or test case to verify the fix”. It seems an agent that knows it has to prove its work is significantly more accurate and less likely to hallucinate a “done” state!

What I Learnt: Behavioral System Design

EWA is built like a Claude agent where it has a brain (orchestrator), reasoning (policy engine), senses (endpoint checker), hands (Jira + Slack tools) and memory (incident store).


Thus, moving beyond simple monitoring, this system creates a truly agentic closed loop: it observes, reasons, decides, acts and remembers, closing the gap between detection and autonomous resolution. This is what differentiates a single prompt from a system that operates.

If designed properly, the orchestrator never does anything directly. It asks tools to observe, asks the policy engine to reason, then dispatches to tools based on the decision. Every component has one job and knows nothing about the others.

Thus, with agentic systems, we start to define goals, shape decision boundaries, orchestrate tools and design workflows. The unit of design has moved from “What does this function do?” to “How does this system behave over time?”. This is very different and is a significant mindset shift.

What I Learnt: The Operational Reality

This is where Agentic AI gets interesting and at the same time risky. They are not just capable but are also more complex to reason about.

What’s Exciting (The Wins)

  • Self-Healing Workflows: Automation of operational tasks where systems can adapt to minor changes instead of simply breaking


  • Engineering Velocity: Drastic reduction in manual intervention for complex, multi-file refactors

What’s Hard (The Risks)

  • Observability & Non-Linear Debugging: Traditional logs don’t help much when an agent enters a logic loop. It becomes difficult to answer: “Why did the agent choose this specific tool at this specific time?” Tracking these non-linear flows requires a completely different observability stack.


  • Guardrails & Cost: Without structural “circuit breakers”, agents can enter recursive loops that transform a technical logic error into a financial one. In an agentic world, unguided autonomy doesn’t just crash a service, it can drain token budgets in minutes.

What I Learnt: The Shift to “Specification of Judgment”

The biggest realization was the shift in our roles: The engineer’s job is becoming the specification of judgment.

We are moving away from writing line-by-line code and towards translating domain knowledge (e.g., Don’t auto-close the Jira ticket on recovery instead leave that to humans), operational experience (e.g., What if the MCP server subprocess hangs instead of failing?) and trust calibrations (e.g., Trust the agent to send Slack alerts without human review: yes) into rules the agent can follow.

Claude handles the execution, but its success depends on our ability to articulate why a system should behave a certain way, not just what it should do. This requires architectural experience to anticipate what could go wrong and the clarity to express those constraints precisely.

Final Thoughts: The Evolution of How We Build

It’s only a matter of time. While the technical risks are real today, the pace of advancement is blistering. We are witnessing a total paradigm shift: we aren’t just writing code anymore, instead we are managing a digital workforce.

For architects, this means rethinking system boundaries. For developers, it means thinking in workflows. I am excited to adapt! This isn’t the evolution of standard coding but the evolution of how we build.

.Sandeep Mewara Github

News Update
Tech Explore
Trend
samples GitHub Profile Readme
Learn Machine Learning with Examples
Machine Learning workflow
What is Dynamic Programming
The Great Inversion: Why AI is Moving from Cloud to Desktop
Explore or build on the Project available here: [Github Link]

Why ‘Service as Software’ is the Industry’s Next Big Bet

I recently caught a presentation by Intuit CTO Alex Balazs, where he described their evolution from a “Do-It-Yourself” software company to an AI-driven expert platform. During the talk, he used a phrase that immediately clicked for me: “Service as Software”.

service-as-software


It was one of those “aha” moments that forced me to pause and re-evaluate the trajectory of the entire SaaS industry. We’ve spent the last twenty years perfecting Software as a Service, but flipping that phrase to Service as Software implies a much deeper shift in how we deliver value. It provoked me to dig into why this isn’t just a trend, but a directional necessity for the next generation of tech.

The Shift: From Passive Tools to Active Experts

For years, the gold standard has been the “System of Record“. We built beautiful digital filing cabinets and powerful calculators, but they were ultimately passive tools. Whether it was an accounting suite or a CRM, the software only provided value if a human expert sat behind the keyboard to drive it. In that model, the value only scales as fast as the person at the controls.

Now, “Service as Software” represents a move toward a “System of Action“. With the rise of agentic AI, software is moving from the “medium” to the “expert.” Recent 2025 research from Capgemini highlights that we are moving beyond “Copilots” to “Agents” where AI that doesn’t just suggest actions but possesses the autonomy to execute end-to-end business processes.

  • SaaS (The Tool): The software provides the interface where the user performs the labor.
  • Service as Software (The Outcome): The software acts as an autonomous agent navigating complexity, identifying optimizations and executing tasks on the user’s behalf.

Why this is the Industry’s Directional Need

As I look at the landscape from a leadership perspective, this shift feels inevitable. We are hitting a ceiling with traditional models for a few key reasons:

  • Solving for “SaaS Fatigue”: The “per-seat” model is under pressure. According to 2026 SaaS pricing forecasts, nearly 60% of enterprise SaaS solutions are shifting toward hybrid or outcome-based pricing. Customers are tired of managing dozens of tools that require constant human attention. They want problems solved, not more licenses to manage.


  • Bridging the Expertise Gap: We are facing a documented global shortage of human experts in complex fields like finance, specialized engineering and data science. By “coding” that expertise directly into the software, we make high-level results accessible at a scale that human labor simply cannot match.


  • Accelerating Time-to-Value: Traditional software often has a long “time-to-value” during onboarding, a period where 63% of customers are already deciding whether to churn. A service-oriented model flips this. By having the software perform the initial heavy lifting for the user, you deliver the “aha moment” almost instantly.

Navigating the Transition: A Technical Leader’s View

Transitioning to this model is an architectural marathon. You don’t just “add AI” and call it a service. It requires a fundamental rethink of the stack.

navigating-transition

  • The “Human-in-the-Loop” Bridge: Trust is the primary hurdle. Successful transitions will likely use a hybrid model where AI performs 80% of the work, but human experts remain available for the “gray areas”. This builds the user’s confidence in the system’s autonomy while maintaining a safety net.


  • Codifying Logic, Not Just Features: We have to shift from building “buttons” to building “agents”. This requires robust reasoning engines that can handle exceptions and ambiguity without breaking.


  • The Observability Mandate: If the software is performing the service, it cannot be a black box. As architects, we must build in deep transparency providing “reasoning logs” so users can always audit why a specific decision was made.

Closing Thoughts

We are moving away from providing digital tools and toward providing digital results. The most successful companies of the next decade won’t just be selling software but they’ll be selling outcomes and confidence.

The transition from being a vendor of tools to being a partner in results is a massive challenge, but for those of us in technical leadership, it’s easily the most exciting problem to solve in a long time. It’s no longer about what our users can do with our software but it’s about what our software can do for our users.

Sandeep Mewara Github
News Update
Tech Explore
Data Explore
samples GitHub Profile Readme
Learn Machine Learning with Examples
Machine Learning workflow
What is Dynamic Programming
The Great Inversion: Why AI is Moving from Cloud to Desktop


The Great Inversion: Why AI is Moving from Cloud to Desktop

For the better part of a decade, the desktop was largely relegated to a passive terminal, a mere high-resolution viewport for remote cloud services. As the industry mantra shifted to “Cloud-First”, local hardware was often treated as an underutilized abstraction.

desktop-ai-back-great-inversion

However, we are now witnessing The Great Inversion. As AI workloads navigate the practical limits of cloud latency, data privacy and operational costs, the center of gravity is visibly shifting back to the local system. We are moving towards the era of the AI-Native Desktop, where the local machine is no longer just a window to the cloud, but is increasingly becoming the primary engine of intelligence.

The Evolution of the “SaaS Margin”

A primary driver of this shift appears to be fundamental economics. Throughout 2024 and 2025, as software providers integrated Large Language Models (LLMs) into their web platforms, it became clear that inference costs could significantly erode margins. This “Token Tax” has encouraged a strategic reckoning across the industry.

  • The Data: According to early 2025 fiscal reports from major SaaS players, AI-related compute costs increased OpEx by an average of 25-30% year-over-year.

     

  • The Cost Shift: Industry analysis from Deloitte and various independent reports suggests that local NPU inference can reduce AI operational costs by up to 90% ( Medium/Vygha, 2025). By migrating specific compute tasks to the desktop, we can transition from a variable OpEx model towards a more sustainable fixed hardware model.

The Proliferation of the AI PC

The “Inversion” is physically supported by a massive hardware refresh. We are no longer designing for underpowered machines. As of Q1 2026, the “AI PC” has moved from a premium category to the industry baseline.

  • The Benchmark: The AI PC has evolved from a niche offering into an enterprise standard. Gartner reports that AI PCs now account for over 55% of all shipments, with nearly 100% of new enterprise purchases featuring dedicated NPUs (Gartner, 2025).


    Microsoft introduced “Copilot+ PCs” as a new Windows category built around local AI acceleration (NPUs) and has continued to expand GA AI features (some in preview) across this category, emphasizing on-device experiences.
     

  • Silicon Supremacy: Standard workstations now ship with 40+ TOPS (Trillion Operations Per Second) capability. This allows for real-time local inference that was previously technically out of reach (Microsoft Learn, 2025).


    Chip vendors are also directly pushing the “on-device inference” narrative as a foundational shift (cost, latency, privacy, reliability).

Compliance and the “Privacy Moat”

Regulatory considerations are making the cloud a complex environment for sensitive data. With the EU AI Act entering its critical enforcement phase in August 2026, there is a clear directional pull toward “Zero-Export” AI solutions (EU AI Act Guide, 2026 ).

  • Apple’s Blueprint: Apple has helped standardize this approach with Apple Intelligence and Private Cloud Compute. Their architecture ensures that if a task can be processed on-device (via the M4’s 38-TOPS Neural Engine), it remains local. Only when necessary does it move to “stateless” servers designed to process data without storing it  (Apple Privacy, 2025 ).

     

  • Data Sovereignty: Modern desktop apps can index a user’s local files to provide personalized AI insights (Local RAG, i.e. Retrieval-Augmented Generation) without ever exposing that intellectual property to a third-party cloud provider. Local-first patterns are re-emerging because they improve resilience and user trust (data control, offline capability, graceful sync).

Performance: Breaking the Latency Wall

The browser is naturally limited by the “spinning wheel” of network latency. For the next generation of Agentic AI, tools that actively assist by observing screen context and reacting in real-time, the network round-trip is often a bottleneck.

cloud-ai-2-local-desktop

Feature

Web App (Cloud AI)

AI-Native Desktop App (NPU)

Response Latency

200ms – 500ms lag

<20ms (Instant)

Data Privacy

Encrypted in Transit

Zero-Export (Stays on Disk)

Offline Capability

Non-existent

Full Functionality

Operational Cost

Per-token / Monthly

One-time Development

System Access

Sandboxed/Limited

Deep File & OS Integration

Moving Forward: The Architect’s Blueprint

To remain competitive in 2026 and beyond, a forward-thinking desktop strategy should aim to capitalize on this hardware-rich environment. While the web remains vital, relying solely on the browser may now carry missed opportunities. A prepared strategy should consider:

  1. Framework Modernization: Exploring lightweight native cores. This involves moving toward Rust-based frameworks like Tauri that interface directly with the local NPU via DirectML or CoreML, rather than relying on memory-heavy wrappers.

     

  2. Hybrid Model Deployment: Integrating Small Language Models (SLMs) like Phi-4 or Llama 3-8B inside the desktop installer. These can handle the majority of daily tasks, reserving the cloud for “Heavy Reasoning” only. 

     

  3. Local Vector Databases: Utilizing local databases (e.g., LanceDB) for hyper-personalized, privacy-first “Long-Term Memory” of the user’s local files, all without requiring a cloud sync.

Conclusion: Towards a Structural Evolution

The current landscape suggests we are moving towards more than just a passing trend. We appear to be entering a structural shift in how software is delivered. There seems to be a renewed potential for the desktop to reclaim its significance, as it offers a compelling intersection where Performance, Privacy and Profit can uniquely align.

However, the most promising products in this new era likely won’t be “desktop-only” in the traditional sense. Instead, there is a clear path for the emergence of desktop-first AI workspaces which will act as platforms that leverage cloud augmentation, sophisticated model-routing and seamless OS integration to redefine the modern workflow.

Final Thought: In 2016, we asked, “Why build a desktop app when you can build a website?” In 2026, the question is increasingly, “Why would a user trust a website with their data when their desktop can do it better, faster and more securely?”

AI seems to be shifting software architecture toward hybrid local-cloud models, which is beginning to elevate the strategic importance of desktop environment once again.

Sandeep Mewara Github
News Update
Tech Explore
Data Explore
samples GitHub Profile Readme
Learn Machine Learning with Examples
Machine Learning workflow
Word Ladder solution
What is Dynamic Programming

Disclaimer: The views and opinions expressed in this article are strictly my own and reflect my personal belief in current market directions. They do not constitute professional or investment advice. Technology landscapes change rapidly, therefore, readers should perform their own due diligence and assess their specific needs before making any architectural or business decisions. I shall not be held responsible for any actions taken based on the contents of this post.

Quick look into Machine Learning workflow

Before we jump into various ML concepts and algorithms, let’s have a quick look into basic workflow when we apply Machine Learning to a problem.

ml-header

A short brief about Machine Learning, it’s association with AI or Data Science world is here.

How does Machine Learning help?

Machine Learning is about having a training algorithm that helps predict an output based on the past data. This input data can keep on changing and accordingly the algorithm can fine tune to provide better output.

It has vast applications across. For example, Google is using it to predict natural disasters like floods. A very common use we hear these days in news are usage in Politics and how to attack the demography of voters.

How does Machine Learning work?

Data is the key here. More the data is, better the algorithm can learn and fine tune. For any problem output, there would be multiple factors at play. Some of them would have more affect then others. Analyzing and applying all such findings are part of a machine learning problem. Mathematically, ML converts a problem output as a function of multiple input factors.

Y = f(x)

Y = predicted output
x = multiple factors as an input

How does a typical ML workflow look?

There is a structured way to apply ML on a problem. I tried to put the workflow in a pictorial view to easily visualize and understand it:

ml-workflow

It’s goes in a cycle and once we have some insights from the published model, it goes back into the funnel as learning to make output better.

datascience-process

Roughly, Data scientists spend 60% of their time on cleaning and organizing data

Walk through an example?

Let’s use dataset of Titanic survivors found here and run through basic workflow to see how certain features like traveling class, sex, age and fare are helping us assess survival probability.

Load data from file

Data could be in various format. Easiest is to have it in a csv and then load it using pandas. More details around how to play with data is discussed here.
.

# lets load and see the info of the dataset

titanicdf = pd.read_csv("data-files/titanic.csv")
print(titanicdf.info())
&lt;class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 14 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   pclass     1309 non-null   int64  
 1   survived   1309 non-null   int64  
 2   name       1309 non-null   object 
 3   sex        1309 non-null   object 
 4   age        1046 non-null   float64
 5   sibsp      1309 non-null   int64  
 6   parch      1309 non-null   int64  
 7   ticket     1309 non-null   object 
 8   fare       1308 non-null   float64
 9   cabin      295 non-null    object 
 10  embarked   1307 non-null   object 
 11  boat       486 non-null    object 
 12  body       121 non-null    float64
 13  home.dest  745 non-null    object 
dtypes: float64(3), int64(4), object(7)
memory usage: 143.3+ KB
None
# A quick sample view
titanicdf.head(3)
 pclasssurvivednamesexagesibspparchticketfarecabinembarkedboatbodyhome.dest
011Allen, Miss. Elisabeth Waltonfemale290024160211.3375B5S2NaNSt Louis, MO
111Allison, Master. Hudson Trevormale0.916712113781151.55C22 C26S11NaNMontreal, PQ / Chesterville, ON
210Allison, Miss. Helen Lorainefemale212113781151.55C22 C26SNaNNaNMontreal, PQ / Chesterville, ON
Data cleanup – Drop the irrelevant columns

It’s not always that the data captured is the only data you need. You will have a superset of data that has additional information which are not relevant for your problem statement. This data will work as a noise and thus it’s better to clean the dataset before starting to work on them for any ML algorithm.
.

# there seems to be handful of columns which 
# we might not need for our analysis 
# lets drop all those column that we know 
# are irrelevant
titanicdf.drop(['embarked','body','boat','name',
'cabin','home.dest','ticket', 'sibsp', 'parch'],
axis='columns', inplace=True)

titanicdf.head(2)
 pclasssurvivedsexagefare
011female29211.3375
111male0.9167151.55
Data analysis

There could be various ways to analyze data. Based on the problem statement, we would need to know the general trend of the data in discussion. Statistics Probability Distribution knowledge help here. For gaining insights to understand more around correlations and patterns, data visualization based insights help.
.

# let's see if there are any highly corelated data
# if we observe something, we will remove that 
# one of the feature to avoid bias

import seaborn as sns
sns.pairplot(titanicdf)

# looking at graphs, seems we don't have 
# anything right away to remove.
titanic-graph
Data transform – Ordinal/Nominal/Datatype, etc

In order to work through data, it’s easy to interpret once they are converted into numbers (from strings) if possible. This helps them input them into various statistics formulas to get more insights. More details on how to apply numerical modifications to data is discussed here.
.

# There seems to be 3 class of people, 
# lets represent class as numbers
# We don't have info of relative relation of 
# class so we will one-hot-encode it
titanicdf = pd.get_dummies(titanicdf,
                              columns=['pclass'])


# Lets Convert sex to a number 
from sklearn import preprocessing
le = preprocessing.LabelEncoder()
titanicdf["sex"] = le.fit_transform(titanicdf.sex) 

titanicdf.head(2)
 survivedsexagefarepclass_1pclass_2pclass_3
01029211.3375100
1110.9167151.55100
Data imputation: Fill the missing values

There are always some missing data or an outlier. Running algorithms with missing data could lead to inconsistent results or algorithm failure. Based on the context, we can choose to remove them or fill/replace them with an appropriate value.
.

# When we saw the info, lots of age were missing
# Missing ages values filled with mean age. 
 
titanicdf.loc[ titanicdf["age"].isnull(), "age" ] = 
titanicdf["age"].mean()

# titanicdf.loc[ titanicdf["fare"].isnull(), "fare"] 
# = titanicdf["fare"].mean() 
# => can do but we will use another way 

titanicdf.info()
&lt;class 'pandas.core.frame.DataFrame'>
RangeIndex: 1309 entries, 0 to 1308
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   survived  1309 non-null   int64  
 1   sex       1309 non-null   int64  
 2   age       1309 non-null   float64
 3   fare      1308 non-null   float64
 4   pclass_1  1309 non-null   uint8  
 5   pclass_2  1309 non-null   uint8  
 6   pclass_3  1309 non-null   uint8  
dtypes: float64(2), int64(2), uint8(3)
memory usage: 44.9 KB
# When we saw the info,
# 1 fare was missing
# Lets drop that one record
titanicdf.dropna(inplace=True)
titanicdf.info()

#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 1308 entries, 0 to 1308
Normalize training data

At times, various data in context are of different scales. In such cases, if the data is not normalized, algorithm can induce bias towards the data that has higher magnitude. Eg, feature A value range is 0-10 and feature B range is 0-10000. In such case, even though a small change in magnitude of A can make a difference but if data is not normalized, feature B will influence results more (which could be not the actual case).
.

X = titanicdf
y = X['survived']
X = X.drop(['survived'], axis=1)

# Scales each column to have 0 mean and 1 std.dev
from sklearn import preprocessing
X_scaled = preprocessing.scale(X)
Split data – Train/Test dataset

It’s always best to split the dataset into two unequal parts. Bigger one to train the algorithm and then then smaller one to test the trained algorithm. This way, algorithm is not biased to just the input data and results for test data can provide better picture.
.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = 
                        train_test_split(X_scaled,y)
X_train.info()
&lt;class 'pandas.core.frame.DataFrame'>
Int64Index: 981 entries, 545 to 864
Data columns (total 6 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   sex       981 non-null    int64  
 1   age       981 non-null    float64
 2   fare      981 non-null    float64
 3   pclass_1  981 non-null    uint8  
 4   pclass_2  981 non-null    uint8  
 5   pclass_3  981 non-null    uint8  
dtypes: float64(2), int64(1), uint8(3)
memory usage: 33.5 KB
Run ML algorithm data

Once we have our training dataset ready as per our need, we can apply machine learning algorithms and find which model fits in best.
.

# for now, picking any one of the classifier - KN
# Ignore details or syntax for now
from sklearn.neighbors import KNeighborsClassifier

dtc = KNeighborsClassifier(n_neighbors=5)
dtc.fit(X_train,y_train)
Check the accuracy of model

In order to validate the model, we use test dataset where comparing the predicted value by model to actual data helps us know about ML model accuracy.
.

import sklearn.metrics as met

pred_knc = dtc.predict(X_test)
print( "Nearest neighbors: %.3f" 
      % (met.accuracy_score(y_test, pred_knc)))
Nearest neighbors: 0.817


Voila! With basic workflow, we have a model that can predict the survivor with more than 80% probability.

Download

Entire Jupyter notebook with more samples can be downloaded or forked from my GitHub to look or play around: https://github.com/sandeep-mewara/machine-learning

Currently, it covers examples on following datasets:

  • Titanic Survivors
  • Sci-kit Iris
  • Sci-kit Digits
  • Bread Basket Bakery

Over time, I would continue building on the same repository with more samples with different algorithms.

Closure

Believe, now it’s pretty clear on how we can attack any problem for an insight using Machine learning. Try out and see for yourself.

We can apply Machine Learning to multiple problems in multiple fields. I have shared a pictorial view of sectors in the AI section that are already leveraging it’s benefit.


Keep learning!

samples GitHub Profile Readme
Learn Python – Beginners step by step – Basics and Examples
Sandeep Mewara Github
Sandeep Mewara Learn By Insight
Matplotlib plot samples
Sandeep Mewara Github Repositories
Learn Microsoft Tech via Videos LiveTV Streams
Machine Learning workflowMicrosoft .NET5

Microsoft AI support for COVID-19

In January 2020, Microsoft launched AI for Health program. Goal is to use Artificial Intelligence (AI) and data to help improve the health of people worldwide. Post COVID-19 hit world wide, Microsoft took steps to support & empower researchers, nonprofits and policymakers with resources.

ai-health

There have been more than 50 million confirmed cases of Covid-19 and more than 1.25 million deaths globally

World Health Organization

Research has accelerated with large-scale computing and open data.

Highlights from Grantees World wide

ai-health-covid-response
Credit: Microsoft

As shared here, few of them are:

  • A grassroots employee volunteer effort at Belgian biopharmaceutical company UCB.
  • UC Riverside researchers utilize quantum-based methods to more accurately predict the effectiveness of proposed Covid-19 inhibitors.
  • IHME, a global health research organization at the University of Washington School of Medicine, forecasts the Covid-19 pandemic.
  • Professor Amanda Randles at Duke University is conducting hundreds of millions of simulations required to help more patients have access to critical ventilators.

Partnership with Organizations

As shared here, few of them are:

  • Partnering with the White House Office of Science and Technology Policy’s (OSTP) High Performance Computing Consortium to support researchers and academia.
  • Working with Brown University’s School of Public Health and the Edmond J. Safra Center for Ethics at Harvard to visualize a common set of measures on testing and risk levels to help everyone know where we are with the pandemic and help policymakers guide their response.
  • Joining University of Oxford and their Government Response Tracker initiative to track and compare government and policy responses to address Covid-19 around the world.

The rapid progress means researchers can more quickly identify potential solutions to combat Covid-19 and provide timely information to policymakers for data-driven decisions that protect communities, cities and regions.

Microsoft blog

Public Information

Using publicly available information from partners, Microsoft have created:

  • a number of interactive visualizations to provide transparency into Covid-19 trends globally.
  • a unique measure called Progress to Zero to help everyone understand, progress in reducing Covid-19 cases, hospitalizations and deaths.
india-progress-to-zero

My Take

There are some great work being done world wide to fight Covid-19 in all possible ways. Organizations like Microsoft are making an active effort to help partners and grantees for Covid-19 response. There is much to do and supports like these will help mankind avert Pandemic.


Keep exploring!

Sandeep Mewara Github
News Update
Tech Explore
Data Explore
samples GitHub Profile Readme
Learn Machine Learning with Examples
Machine Learning workflow
What is Data Science
Word Ladder solution
What is Dynamic Programming
Learn Microsoft Tech via Videos LiveTV Streams Microsoft .NET5

Learn AI ML with Netflix’s Fei Fei!

Off late, Microsoft has been working on providing more and more learning materials. This is as per their Global Skills Initiative aimed at helping 25 million people worldwide acquire new digital skills.

global-initiative

Our goal is to ignite the passion to solve important problems relevant to their lives, families and communities.

Microsoft

Last week, they partnered with Netflix to release a new learning experience featuring a young female hero Fei Fei, who has a passion for science and explores space.

Newly launched

Microsoft has launched three new modules under Explore Space with “Over the Moon” learning path. These modules will help learn basic concepts of data science, artificial intelligence and machine learning:

  1. Plan a Moon Mission using the Python Pandas Library
  2. Predict Meteor Showers using Python and VC Code
  3. Use AI to Recognize Objects in Images using Azure Custom Vision
fei-fei-explore

The movie’s story takes place in a beautifully animated universe and tackles problems real-life space engineers face.

How is it?

They cover the basic workflow of a machine learning problem similar to what I shared in my previous article here.

Exercises also provide a professional experience as they are built on Visual Studio Code and use Azure Cognitive Services.

Looking at it, seems a fun way to learn and explore the data world. Microsoft is really trying to make things simple and available to all. An effort worth to try out. Would recommend and ask to give it a shot.


Keep learning!

samples GitHub Profile Readme
Learn Python – Beginners step by step – Basics and Examples
Sandeep Mewara Github
Sandeep Mewara Learn By Insight
Matplotlib plot samples
Sandeep Mewara Github Repositories
Learn Microsoft Tech via Videos LiveTV Streams
Machine Learning workflow

Dwell in Data Science world

Let’s walk into the Data Science world and see what it means and how does it connect with other terms that we often hear in the context – Artificial Intelligence & Machine Learning.

data-science-explore

What is Data Science?

In a context of a particular domain, data science is gaining insights into data through statistics, computation and visualization. Below diagram represents this multi-focal field well:

data-science-venn

For a real world problem, based on statistics, we make certain assumptions. Based on assumptions, we make a learning model using mathematics. With this model, we make software that can help validate and solve the problem. This leads to solving complex problems fast and more accurately.

How to use Data Science?

Clearly, it can be defined as a process. When followed, it would help move towards destination with proper next steps:

datascience-process
Adapted from: Harvard Data Science lecture slide

There are multiple iterative stages that solves specific queries. Based on answers to the queries, we might have to circle back and reassess the steps of the entire process.

Various stages involved?

Once we have a defined process, it’s easier to break it down into different functional groups. It would help us interpret how to visualize, connect them and know who can help us at each of those stage:

what-is-data-science
Credit: Rubens Zimbres article

There is a strong correlation of business intelligence with data science here. Current advancements in algorithms and tools has helped us improve accuracy for each of the stages above.

Where does AI or ML fits in?

Data Science, Artificial Intelligence & Machine Learning are different but often used interchangeably. There are overlaps and a part covers all of them together:

data-science-overlap

On a high level, part of the data science provides processed data. AI or ML or DL helps to process the data to get the needed output.

Artificial Intelligence (AI)

It is a program that enables machine to mimic human behavior. As such, goal here is to understand human intelligence, learn to imitate and act accordingly. I came across a good AI exhibit by BCG:

ai-bcg-analysis
Credit: BCG Group tweet

Self driving cars and route change suggestion are few common AI solutions

Machine Learning (ML)

These are AI techniques that enables machines to learn from examples, without being explicitly programmed to do so. It incorporates mathematics and statistics to learn from itself and improve with experience.

ml-sample

Recommendation engines & Spam email classification are few common ML solutions

I will cover Machine Learning in much more detail in later posts.

Deep Learning (DL)

These are subset of ML that makes the computation of multi-layer neural network feasible. It does not require feature selection/engineering. They classify information similar to human brains. They often continue to improve as the size of your data increases.

dl-sample

Face detection and number recognition are few common DL solutions

Moving On …

Overall, Data Science is more about extracting insights to build a robust strategy for business solution and is different from AI or ML.

To read more around differences between the Data Science and other terms, Vincent Granville has shared in detail here.

With above, we have entered into the Data Science world. Going forward, I will concentrate more on Machine Learning aspect for now.


Keep exploring!

samples GitHub Profile Readme
Learn Python – Beginners step by step – Basics and Examples
Sandeep Mewara Github
Sandeep Mewara Learn By Insight
Matplotlib plot samples
Sandeep Mewara Github Repositories

Three A’s of the Next Gen World!

There are many new ideas and research going on in today’s tech world. I believe the potential of maximum disruption is by the three A’s – Analytics, Automation & Artificial Intelligence. Because of their scope of evolution and impact, they tend to be the foundation of the next generation tech world.
.

next-gen

A natural progressive way of doing it would be to:

  • first gather data (analytics)
  • define what’s crucial or repetitive that can be replaced to ease the flow (automate)
  • collate such multiple replaced things to move into predictive analysis and actions to complete a complex task (artificial intelligence)

Day by day, it looks more feasible to achieve above because of technology innovations and the processing power.

Analytics

It is the first step towards anything more concrete and is a great supportive to theories and assumptions. Enables to have data backed decisions which are easy to relate and comprehend.

This would mean collecting data and making better use of it. This would need to have a data architecture setup. With it, it would be easier to know and decide what and how to collect. Post it, using any data visualization tool, it would be easier to deduce insights, patterns and plan accordingly.

It’s not about just any or more data, but the right, quality data that would matter

Automation

It is solving for complex processes using technology. Data analytics help a great deal identify and solve for such processes. These repetitive mundane tasks can be automated and the efforts saved from it can be put elsewhere. Thus helps in concentrating on tasks that needs more human attention.

Just like machines brought industrial revolution to factory floor, automation has similar potential to transform most of our day to day work. This could lead to enhanced productivity, thus better outcomes, leading to more accurate predictions and optimizations.

Only way to scale the massive amount of work in defined time without hiring an army of professionals

Artificial Intelligence

This is exploring the world that was considered impossible a decade ago by most of us. It is easier to relate and understand when comparing with human capabilities like recognizing handwriting or identifying a particular disease cell. Not just it, AI has great potential to automate non-routine tasks.

With AI, we can analyze data, understand concepts, make assumptions, learn behaviors and provide predictions at a scale with detail that would be impossible for individual humans. It can also have self-learning to improve with more and more interactions with data and human.

AI evolution is being greatly supported with advanced algorithms and improved computing power & storage.

Potential

There is a nice data backed assessment and shareout by McKinsey Global Institute on the three A’s. I would suggest a must read of it. In there, they shared AI/ML potential usecases across industries.

ai-ml-potential-industries
Credit: McKinsey & Co.

AI and Automation will provide a much-needed boost to global productivity and may help some ‘moonshot’ challenges.

McKinsey Global Institute

Wrap Up

AI combined with various Process Automation and powerful Data Analytics transforms into an intelligent automated solution. Potential of these solutions are huge.

It would be more appropriate to say that it’s no more a choice but compulsion for all sectors to go through this digital transformation. Those who are able to do it will be setup sooner for the next generation of the tech world. It will provide them with an edge over their competitors, putting them in a position to take advantage big time.


Keep exploring!

samples GitHub Profile Readme
Learn Python – Beginners step by step – Basics and Examples
Sandeep Mewara Github
Sandeep Mewara Learn By Insight
Matplotlib plot samples
Sandeep Mewara Github Repositories

Data Science Virtual Conference 2020

Open Data Science Conference (ODSC) is one of the biggest specialized data science upcoming event, scheduled on December 8-9, 2020. Read details about the event here. Theme for the year is RETHINK AI.

Virtual Conference is spread across two days:
– December 8th – ODSC India
– December 9th – ODSC Asia & Pacific

The Largest Applied Data Science & AI Conference Returns to India!

Call for speakers are open for submission. Registration is open to book seat.

data-science-conf-2020

Learn the latest AI & data science topics, tools, and languages from some of the best and brightest minds in the field.

Event highlight shared by ODSC team

The conference promises to accelerate your knowledge on data science and related disciplines with insightful sessions, workshops and speakers from various fields. There would be speakers that include core contributors to many open source libraries and languages.

Happy connecting!

samples GitHub Profile Readme
Learn Python – Beginners step by step – Basics and Examples
Sandeep Mewara Github
Sandeep Mewara Learn By Insight
Matplotlib plot samples
Sandeep Mewara Github Repositories

Probability Distribution – An aid to know the data

A probability distribution helps understand the likelihood of possible values that a random variable can take. It is one of the must needed statistical knowledge for any data science aspirant.

probability-header

Few consider, Probability distributions are fundamental to statistics, like data structures are to computer science

In Layman terms

Let’s say, you pick any 100 employees of an organization. Measure their heights (or weights). As you measure them, create a distribution of it on a graph. Keep height on X-Axis & frequency of a particular height on Y-Axis. With this, we will get a distribution for a range of heights.

This distribution will help know which outcomes are most likely, the spread of potential values, and the likelihood of different results.

Basic terminology

Random Sample

The set of 100 people selected above in our example will be termed as random sample.

Sample Space

The range of possible heights of the 100 people is our sample space. It’s the set of all possible values in the setup.

Random Variable

The height of the 100 people measured are termed as random variable. It’s a variable that takes different values of the sample space randomly.

Mean (Expected Value)

Let’s say most of the people in those 100 are of height 5 feet, 3 inches (making it an average height of those 100). This would be termed expected value. It’s an average value of a random variable.

Standard deviation & Variance

Let’s say most of the people in those 100 are of height 5 feet, 1 inches to 5 feet, 5 inches. This is variance for us. It’s an average spread of values around the expected value. Standard Deviation is the square root of the variance.

Types of data
  • Ordinal – They have a meaningful order. All numerical data fall in this bucket. They can be ordered in relative numerical strength.
  • Nominal – They cannot be ordered. All categorical data fall in this bucket. Like, colors – Red, Blue & Green – there cannot be an order or a sequence of high or low in them by itself.
  • Discrete – an ordinal data that can take only certain values (like soccer match score)
  • Continuous – an ordinal data that can take any real or fractional value (like height & weight)

In Continuous distribution, random variables can have an infinite range of possible outcomes

Probability Distribution Flowchart

Following diagram shares few of the common distributions used:

Based on above diagram, will cover three distributions to have a broad understanding:

Uniform Distribution

It is the simplest form of distribution. Every outcome of the sample space has equal probability to happen. An example would be to roll a fair dice that would have an equal probability outcome of 1-6.

uniform-distribution
Normal (Gaussian) Distribution

The most common distribution. Few would recognize this by a ‘bell curve’. Most values are around the mean value making the distribution arrangement symmetric.

Central limit theorem suggests that sum of several independent random variables is normally distributed

normal-distribution

The area under the distribution curve is equal to 1 (all the probabilities must sum up to 1)

A parameter Mew drives the distribution center (mean). It corresponds to the maximum height of the graph. A parameter Sigma corresponds to the range of variation (variance or standard deviation).

standard-normal
Credit: Wikipedia

68–95–99.7 rule (empirical rule) – approximate percentage of the data covered by ranges defined by 1, 2, and 3 standard deviations from the mean

Exponential Distribution

It is where a few outcomes are most likely with a rapid decrease in probability to all other outcomes. An example of it would be a car battery life in months.

exponential-graph

A parameter Beta deals with scale that defines the mean and standard deviation of the distribution. A parameter Lambda deals with rate of change in the distribution

Probability Distribution Choices

I came across an awesome representation of the probability distribution choices. It works as a cheat sheet to understand the provided data.

Wrap Up

Though above is just an introduction, believe it should be good enough to start, correlate and understand some basics of machine learning algorithms. There would be more to it while working on algorithms and problems while analyzing data to predict trends, etc.


Keep learning!

samples GitHub Profile Readme
Learn Python – Beginners step by step – Basics and Examples
Sandeep Mewara Github
Sandeep Mewara Learn By Insight
Matplotlib plot samples