Friday, January 31, 2014

Ye na thi hamari qismat

Ye na thi hamari qismat ke visal e yar hota,
Agar aur jeete rehte, yehi intezar hota !

Tere vaade pe jiye hum, to ye jan jhoot jana,
Ke khushi se mar na jatay agar eitebar hota !
Teri nazuki se jana ke bandha tha ehd buoda,
Kabhi tu na tor sakta, agar astawaar hota !

Ye kahan ki dosti hai, ke bane hain dost naasih,
Koi chara saaz hota, koi gham gusaar hota !

Kahon kis se main ke kia hai, shab e gham buri bala hai,
Mujhe kia bura tha marna, agar eitebar hota !

Hoye mar ke hum jo ruswa, hoye kyon na gharq e darya,
Na kabhi janaza uthta, na kaheen mazaar hota !

Ye masa'il e tasawwuf, ye tera bayan Ghalib,
Tujhe hum wali samajhte jo na bada khuwar hota !

Spring Integration concepts

Channel Adapter : One way message

Gateway : Bidirectional message

Message: Unit of info passed between two endpoints. Has header and payload.

Channel: Decouples producer from consumer.



1.2. Spring Integration’s support for enterprise integration patterns

Enterprise Integration Patterns describes the patterns used in the exchange of messages, as well as the patterns that provide the glue between applications. Like the diagram in figure 1.1, it’s about messaging and integration in the broadest sense, and the patterns apply to both intra-application and inter application scenarios. Spring Integration supports the patterns described in the book, so we need to establish a broad understanding of the definitions of these patterns and the relations between them.
From the most general perspective, only three base patterns make up enterprise integration patterns: MessageMessage Channel, and Message EndpointFigure 1.2 shows how these components interact with each other in a typical integration application.
Figure 1.2. A message is passed through a channel from one endpoint to another endpoint.
There are two main ways to differentiate between these patterns. First, each pattern has more specific subtypes, and second, some patterns are composite patterns. This section focuses on the subtypes so you have a clear understanding of the building blocks. Composite patterns are introduced as needed throughout the book.

1.2.1. Messages

A message is a unit of information that can be passed between different components, called message endpoints. Messages are typically sent after one endpoint is done with a bit of work, and they trigger another endpoint to do another bit of work. Messages can contain information in any format that’s convenient for the sending and receiving endpoints. For example, the message’s payload may be XML, a simple string, or a primary key referencing a record in a database. See figure 1.3.
Figure 1.3. A message consists of a single payload and zero or more headers, represented here by the square and circle, respectively.
Each message consists of headers and a pay-load. The header contains data that’s relevant to the messaging system, such as theReturn Address or Correlation ID. The payload contains the actual data to be accessed or processed by the receiver. Messages can have different functions. For example, a Command Message tells the receiver to do something, an Event Message notifies the receiver that something has happened, and a Document Message transfers some data from the sender to the receiver.
In all of these cases, the message is a representation of the contract between the sender and receiver. In some applications it might be fine to send a reference to an object over the channel, but in others it might be necessary to use a more interoperable representation like an identifier or a serialized version of the original data.

1.2.2. Message Channels

The message channel is the connection between multiple endpoints. The channel implementation manages the details of how and where a message is delivered but shouldn’t need to interact with the payload of a message. Whereas the most important characteristic of any channel is that it logically decouples producers from consumers, there are a number of practicalimplementation options. For example, a particular channel implementation might dispatch messages directly to passive consumers within the same thread of control. On the other hand, a different channel implementation might buffer messages in a queue whose reference is shared by the producer and an active consumer such that the send and receive operations each occur within different threads of control. Additionally, channels may be classified according to whether messages are delivered to a single endpoint (point-to-point) or to any endpoint that is listening to the channel (publish-subscribe). As mentioned earlier, regardless of the implementation details, the main goal of any message channel is to decouple the message endpoints on both sides from each other and from any concerns of the underlying transport.
Two endpoints can exchange messages only if they’re connected through a channel. The details of the delivery process depend on the type of channel being used. We review many characteristics of the different types of channels later when we discuss their implementations in Spring Integration. Message channels are the key enabler for loose coupling. Both the sender and receiver can be completely unaware of each other thanks to the channel between them. Additional components may be needed to connect services that are completely unaware of messaging to the channels. We discuss this facet in the next section on message endpoints.
Channels can be categorized based on two dimensions: type of handoff and type of delivery. The handoff can be either synchronous or asynchronous, and the delivery can be either point-to-point or publish-subscribe. The former distinction will be discussed in detail in the synchronous versus asynchronous section of the next chapter. The latter distinction is conceptually simpler, and central to enterprise integration patterns, so we describe it here.
In point-to-point messaging (see figure 1.4), each single message that’s sent by a producer is received by exactly one consumer. This is conceptually equivalent to a postcard or phone call. If no consumer receives the message, it should be considered an error. This is especially true for any system that must support guaranteed delivery. Robust point-to-point messaging systems should also include support for load balancing and failover. The former would be like calling each number on a list in turn as new messages are to be delivered, and the latter would be like a home phone that’s configured to fall back to a mobile when nobody is home to answer it.
Figure 1.4. A Point-to-Point Channel
As these cases imply, which consumer receives the message isn’t necessarily fixed. For example, in the Competing Consumers(composite) pattern, multiple consumers compete for messages from a single channel. Once one of the consumers wins the race, no other consumer will receive that message from the channel. Different consumers may win each time, though, because the main characteristic of that pattern is that it offers a consumer-driven approach to load balancing. When a consumer can’t handle any more load, it stops competing for another message. Once it’s able to handle load again, it will resume.
Unlike point-to-point messaging, a Publish-Subscribe Channel (figure 1.5) delivers the same message to zero or more subscribers. This is conceptually equivalent to a newspaper or the radio. It provides a gain in flexibility because consumers can tune in to the channel at runtime. The drawback of publish-subscribe messaging is that the sender isn’t informed about message delivery or failure to the same extent as in point-to-point configurations. Publish-subscribe scenarios often require failure-handling patterns such asIdempotent Receiver or Compensating Transactions.
Figure 1.5. A Publish-Subscribe Channel

1.2.3. Message endpoints

Message endpoints are the components that actually do something with the message. This can be as simple as routing to another channel or as complicated as splitting the message into multiple parts or aggregating the parts back together. Connections to the application or the outside world are also endpoints, and these connections take the form of channel adapters, messaging gateways, or service activators. We discuss each of them later in this section.
Message endpoints basically provide the connections between functional services and the messaging framework. From the point of view of the messaging framework, endpoints are at the end of channels. In other words, a message can leave the channel successfully only by being consumed by an endpoint, and a message can enter the channel only by being produced by an endpoint. There are many different types of endpoints. We discuss a few of them here to give you a general idea.
Channel Adapter
Channel Adapter (see figure 1.6) connects an application to the messaging system. In Spring Integration we chose to constrict the definition to include only connections that are unidirectional, so a unidirectional message flow begins and ends in a channel adapter. Many different kinds of channel adapters exist, ranging from a method-invoking channel adapter to a web service channel adapter. We go into the details of these different types in the appropriate chapters on different transports. For now, it’s sufficient to remember that a channel adapter is placed at the beginning and the end of a unidirectional message flow.
Figure 1.6. Channel Adapter
Messaging Gateway
In Spring Integration, a Messaging Gateway (see figure 1.7) is a connection that’s specific to bidirectional messaging. If an incoming request needs to be serviced by multiple threads but the invoker needs to remain unaware of the messaging system, an inbound gateway provides the solution. On the outbound side, an incoming message can be used in a synchronous invocation, and the result is sent on the reply channel. For example, outbound gateways can be used for invoking web services and for synchronous request-reply interactions over JMS.
Figure 1.7. Messaging Gateway
A gateway can also be used midstream in a unidirectional message flow. As with the channel adapter, we’ve constrained the definition of messaging gateway a bit in comparison to Enterprise Integration Patterns (see figure 1.8.)
Figure 1.8. Messaging Gateway and Channel Adapters
Service Activator
Service Activator (see figure 1.9) is a component that invokes a service based on an incoming message and sends an outbound message based on the return value of this service invocation. In Spring Integration, the definition is constrained to local method calls, so you can think of a service activator as a method-invoking outbound gateway. The method that’s being invoked is defined on an object that’s referenced within the same Spring application context.
Figure 1.9. Service Activator
Router (see figure 1.10) determines the next channel a message should be sent to based on the incoming message. This can be useful to send messages with different payloads to different, specialized consumers (Content-Based Router). The router doesn’t change anything in the message and is aware of channels. Therefore, it’s the endpoint that’s typically closest to the infrastructure and furthest removed from the business concerns.
Figure 1.10. Router
Splitter (see figure 1.11) receives one message and splits it into multiple messages that are sent to its output channel. This is useful whenever the act of processing message content can be split into multiple steps and executed by different consumers at the same time.
Figure 1.11. Splitter
An Aggregator (figure 1.12) waits for a group of correlated messages and merges them together when the group is complete. The correlation of the messages typically is based on a correlation ID, and the completion is typically related to the size of the group. A splitter and an aggregator are often used in a symmetric setup, where some work is done in parallel after a splitter, and the aggregated result is sent back to the upstream gateway.
Figure 1.12. Aggregator
You’ll see many more patterns throughout the book, but what we covered here should be sufficient for this general introduction. If you paid close attention while reading the first paragraph in section 1.2, you may have noticed that we said Spring Integrationsupports the enterprise integration patterns, not that it implements the patterns. That’s a subtle but important distinction. In general, software patterns describe proven solutions to common problems. They shouldn’t be treated as recipes. In reality, patterns rarely have a one-to-one mapping to a single implementation, and context-dependent factors often lead to particular implementation details.
As far as the enterprise integration patterns are concerned, some, such as the message and message channel patterns, are more or less implemented. Others are only partially implemented because they require the addition of some domain-specific logic; examples are the content-based router in which the content is dependent on the domain model and the service activator in which the service to be activated is part of a specific domain. Yet other patterns describe individual parts of a larger process; examples are the correlation ID we mentioned when describing splitter and aggregators and the return address that we discuss later. Finally, there are patterns that simply describe a general style, such as the pipes-and-filters pattern. With these various pattern categories in mind, let’s now see how the concept of inversion of control applies to Spring Integration’s support for the patterns.

Of the two, synchronous communication is more straightforward: the recipient of the call is known in advance, and the message is received immediately (see figure 2.4). The invocation, processing, and response occur in the same thread of execution (like a Java thread if the call is local or a logical thread if it’s remote). This allows you to propagate a wealth of contextual information, the most common being the transactional and security context. Generally, the infrastructure required to set it up is simpler: a method call or a remote procedure call. Its main weaknesses are that it’s not scalable and it’s less resilient to failure.
Figure 2.4. Synchronous message exchange: the message is received immediately by the provider.
Scaling up is a problem for synchronous communication because if the number of simultaneous requests increases, the target component has few alternatives, for example:
  • Trying to accommodate all requests as they arrive, which will crash the system
  • Throttling some of the requests to bring the load to a bearable level[2]
    2 Throttling is the process of limiting the number of requests that a system can accommodate by either postponing some of them or dropping them altogether.
When the load increases, the application will eventually fail, and you can do little about it.
The lack of resilience to failure comes from the fundamental assumption that the service provider is working properly all the time. There’s no contingency, so if the service provider is temporarily disabled, the client features that depend on it won’t work either. The most obvious situation is a remote call that fails when the service provider is stopped, but this also applies to local calls when an invoked service throws a RuntimeException.
Table 2.1. Synchronous and asynchronous communication compared
DefinitionRequest is delivered immediately to provider, and the invoker blocks until it receives a response.The invoker doesn’t wait to receive a response. Requests are buffered and will be processed when the provider is ready.
  • Simple interaction model
  • Immediate response
  • Allows the propagation of invocation context (transaction, security)
  • Good scalability under load
  • Resilience to failure (requester and provider needn’t be available at the same time)
  • Lack of resilience to failure
  • Doesn’t scale properly under load
  • Complex interaction model due to increased concurrency
  • Hard to debug

Service Activator Endpoint Example

A Service Activator is a generic endpoint that invokes a method on a bean when a message arrives at an input channel. Declaration of the endpoint is shown in the following snippet:

// bean to be invoked
The service-activator endpoint picks up a message as soon as it arrives on the positions-channel and calls the processNewPosition method on the bean. Any code required to fetch messages or invoke the bean’s method is already included in the service-activator snippet. This is the power of the framework.

Priority Channel

The PriorityChannel is a subclass of QueueChannel with just one additional characteristic—prioritization of messages. If you need to send a high-priority message immediately, then PriorityChannel is the one to use. The easiest way is to set the PRIORITY property on the MessageHeader when creating a message.
Let’s look at an example to create a message with priority. The publishPriorityTrade method publishes a new Trade onto the provided channel. Priority of the message is set by using the MessageHeader’s PRIORITY property. Its value is an integer, thus the higher the value, the higher the priority.
public void publishPriorityTrade(Trade t) {
  Message tradeMsg = MessageBuilder.withPayload(t).
    setHeader(MessageHeades.PRIORITY, 10).build();

  priorityChannel.send(tradeMsg, 10000);

  System.out.println("The Message is published successfully");
Messages with a higher priority will end up at the top of the queue, while the lower-priority messages will be pushed down. The default behavior is to useMessageHeaders’ PRIORITY property to sort the messages.
In order to create a priority channel, use the priority-queue element as shown in the following XML code:

The priority-queue tag lets the framework create a PriorityChannel with a given capacity.
If you need to further customize priorities, you need to provide your own comparator by implementing Comparator> to the constructor. The following code snippet shows the AccountComparator:
public class AccountComparator implements Comparator> {

  public int compare(Message msg1, Message msg2) {
    Account a1 = (Account)msg1.getPayload();
    Account a2 = (Account)msg2.getPayload();

    Integer i1 = a1.getAccountType();
    Integer i2 = a2.getAccountType();

    return i1.compareTo(i2);
Once you define the Comparator, you need to let the framework know you are going to use it for all the messages coming into the priority channel. You do this by using the comparator tag:

The priority-queue expects capacity and comparator values. In the above snippet, we set the AccountComparator as the comparator on the channel.

PublishSubscribe Channel

Use PublishSubscribeChannel if you need to send a message to multiple consumers. This is the implementation of the SubscribableChannel interface out of the box. There are no receive methods in this channel because the message reception is handled by a subscriber called MessageHandler.
The declaration of the channel in an XML config file is simple and straightforward:

Spring Integration provides endpoints such as Service Activators, Channel Adapters, Message Bridges, Gateways, Transformers, Filters, and Routers.

Thursday, January 30, 2014

The Lean Startup by Eric Ries - Notes

Process to turn product insights into a great company?

Is determination, brilliance, great timing and above all great product is the mantra for fame and fortune?

Often a promising start leads to failure.
Most startups fail
Most new products are not successful
Most new ventures do not live up to their potential

Argues - assumption if we build it, they will come. When we fail, we have a ready made excuse:
 1. We didn't have the right stuff
 2. We weren't visionary enough
 3. Weren't in the right place at the right time.

Author rejects the above line of thinking and argues its the boring stuff what matters the most.
 ** Entrepreneurship is a kind of management. i.e. its dull, serious and bland.

IMVU model AKA Lean Startup model
Instead of spending years perfecting the product. Build a minimum viable product, an early product that is terrible, full of bugs and crash-your-computer yes really stabilty problems. Then ship it to customers way before its ready. And lastly ship/change product every day. Customer feedback was viewed as only one source of information about the product and overal vision. Preferred running experiments on the customers than to cater to their whims.

LS model is build on lean manufacturing, design thinking, customer development and agile development.
** LS represents new approach to creating continous innovation.

LS is catagorized by an
 1. extremely fast cycle time
 2. focus on what customer wants (without asking them)
 3. scientific approach to making decisions.

When products fail, engineers view these as technical problems that require technical solutions
 1. Better architecture
 2. Better engineering process
 3. Better discipline
 4. Focus
 5. Product vision
** These suppose fixes let to still more failures.

Business and marketing functions of a startup should be considered as important as engineering and product development and therefore deserve an equally rigorous methodology to guide them. (Customer Development)
  -- Steve blank

Lean Manufacturing - Innovated at Toyota is the bases for LS.

Lean Startup's five principles
 1. Entrepreneurs are everywhere   (Big org, garage)
 2. Entrepreneurship is management (Startup is institution not just product)
 3. Validated learning (Startups exists to learn how to build sustainable business)
 4. Build-Measure-Learn  (Idea-product, customer response-pivot/perservere)
 5. Innovation Accouting (hold innovators accountable)

 Why Startups Fail
  1. Allure of good plan, solid strategy and thorough market research.
     In earlier eras, these things were indicators of likely success.
     But startups work under uncertainty (don't know customers/product)

  ** More uncertainties mean un predictability.
  ** Chaos is not an answer.

 Book is divided into 3 sections.
 Vision     - gauge if they are making progress
 Steer      - Method to decide pivot/presevere
 Accelerate - Power of small batches.

Dev ops functional spec notes

Dev ops

Repository - a collection of binary software artifacts & metadata stored in a defined directory structure.
e.g - maven, ivy, mercury,
Maven - can store jar, ware, zip etc

Release Artifact  - never change, static artifacts created by specific versioned release.
Snapshot Artifact - generated during development, has both version and timestamp in its name.

Coordinates - for locating artifcats. e.g. maven (groupId, artifactId, version, packaging). Coordinates are translated into url.

List of build automation softwares

| Source Control   |   Build Tools   | Continous Integration |
| SVN               |   Ant                   | Jenkins/Hudson           |
| Mercury          |   Maven              | AnthillPro                   |
| Perforce          |   Gradle              | Bamboo                      |
| Git                  |                            | Cruise Control            |
| CVS               |                            | BuildWeb                   |

John Allspaw and Paul Hammond of Flickr Presentation on Dev Ops.
Key points.
Traditional thinking - Dev - adds feature, Ops - keep site stable and fast
Argues - Ops job is to enable the business NOT to keep the site stable and fast, and this is Dev job too.
Business requires change! But change is the root cause of outages!
Choice - discourage change to keep business running or allow?
Lower risk with tools & culture.
Tools - Automated infrastructure (chep, puppet, AppD etc). One step built and deploy
Culture - Respect, Don't hide things, Trust, Healthy attitude towards failure, Avoid blame.

Wednesday, January 29, 2014

Abu Ubaydahs wisdom

Nothing is more fulfilling than knowledge.
Nothing is more profitable than tolerance
Nothing is more valuable than din.
No friend is more beautiful than intelligence
No companion is more evil than ignorance
Nothing is more esteemed than piety
Nothing is more satisfying than leaving base desires
No deed is better than pondering
No good is higher than patience
No evil is worse than pride
No messenger is more just than truth
No guide is better advisor than honesty
No poverty is more low than greed
No wealth is more ill fortuned than hoarding
No life is superior than good health
No lifestyle is more enjoyable than chastity
No worship is higher than humility
No asceticism is preferable to contentment
No guard is more protecting than silence
No absent thing is closer than death.

Tuesday, January 28, 2014

Notes from Agile Estimating and Planning Works by Mike Cohn

Notes from Agile Estimating and Planning Works by Mike Cohn

Purpose of the book - Agile approach to estimating and planning.

Part 1: The Problem and the Goal

** Important to understand purpose of planning
“Planning is everything. Plans are nothing.”
—Field Marshal Helmuth Graf von Moltke

Plans guide our investment decisions. e.g. If a project takes 6 months and cost 1 mil we might accept it compare to 2yrs and 4 mil.

Plans outlines resource needed, track to deliverables. Without plan projects go through any number of problems

Yet planning is difficult and often plans are wrong
Two extremes - No plan or over plan and over confidence.

Without a plan cannot answer basic questions (when will the work be done)
Over plan -

Plan are off 60 - 160%
i.e. 20 weeks estimate means 17-23 weeks

Planning is to find optimal solution

Why plan then if its difficult?
A good plan supports - reduce risk, uncertainty, better decision, trust and info

Plans iterate. If no one comes with a better idea later it fails.

good plan? - reliable for making decisions

Agile planning - willing to change

Planning is more imp than plan **

** Estimates given early are less accurate than given later.

1.This chapter started by making the claim that overplanning and doing no planning are equally dangerous. What is the right amount of planning on your current project?
2.What other reasons can you think of for planning?
3.Think of one or two of the most successful projects in which you have been involved. What role did planning play in those projects?
** Nearly two-thirds of projects significantly overrun their cost estimates (Lederer and Prasad 1992)
** 64% of the features included are rarely or never used (Johnson 2002)
** The avg project exceeds its schedule by 100% (Standish 2001)

Why planning fails?
1. Planning is by activity rather than feature
2. Multi tasking
3. Priority
4. Ignoring uncertainty
5. Estimates becomes commitments

Discussion Questions

1.What problems result from plans being based on activities rather than deliverable features?
2.In your current environment, is an estimate the same as a commitment? What problems does this cause? What could you do to change this misperception?
3.In what ways does multitasking affect your current project? How could you reduce that impact?
“A good plan violently executed now is better than a perfect plan executed next week.”
—General George S. Patton
Authors of Agile Manifesto values
1. Individuals and interactions over process and tools
2. Working software over doc
3. Customer collab over negotations
4. Responding to change over plan

Agile approach to projects
1. Work as one team
2. Work in short iterations
3. Deliver something each iteration
4. Focus on business priorities
5. Inspect and adapt

Agile approach to planning
The planning onion. Agile teams plan at least at the release, iteration, and day levels.

1.How would working as a unified whole team have affected your current or last project?
2.What are the conditions of satisfaction on your current project? Do all project stakeholders and participants agree on all of them? What risks are there to proceeding on a project that does not have agreement on all conditions of satisfaction?
3.Why are budget and schedule listed in Figure 3.2 as conditions of satisfaction to be considered during release planning but not during iteration planning?
Part 2: Estimating Size


Separate estimate of size from duration
Story points are relative - define a small unit of work and use it as metric
Ideal time is not equal to elapsed time - (Since ppl take sick time, meetings etc)
Almost estimate on ideal time.
Not a good idea to divide estimates based on skills or area
Estimates are done by all irrespective who works on it

Estimate stories using fibonacci 1, 2, 3, 5, 8, small stories are ok to count as 0.
Epics and Themes constitute 13, 20, 40, 100 sizes.

Estimate technique
1. Ask the expert.
2. Analogy (Relative)
3. Disaggregate (split)

Planning poker.

Velocity is a great equalizer

When to reestimate - after learning one of the similar story has drifted
What to do?
No reestimate
Switch the velocity
re-estimate with new size

re-estimating partially completed stories

Story points vs Ideal days -- Prefer story points

Part 3: Planning for Value
Prioritize --> value, cost, new knowledge, risk
“The indispensable first step to getting what you want is this: Decide what you want.”
—Ben Stein
e.g. building security framework into app --> value not much, cost low initially high later, new knowledge none, risk high coz it adds no value to the product success.
e.g. 2 --> user interface, value much, cost low initially, knowledge yes, risk low coz it adds lot of value.

Financial prioritization
“As a general rule of thumb, when benefits are not quantified at all, assume there aren’t any.”
—Tom DeMarco and Timothy Lister
Some projects are done for generating revenue some to cut cost, if we can estimate we can plan accordingly.

Source of return
1. new revenue
2. incremental revenue
3. retained revenue
4. operational efficiencies

Prioritization Desirability
“If you have a choice of two things and can’t decide, take both.”
—Gregory Corso

when to split story - when it doesn't fit in iteration or smaller stories will allow accurate estimates.

How to split stories?
1. around data boundaries
2. operation boundaries
3. cross cutting concerns
4. performance
5. don't split on tasks
6. avoid temptation of related changes

Part 4 : Scheduling

Release planning - covers longer period than iteration.
It tells how much to develop and how long
acts as guide post

Part 5: Tracking and Communicating

Part 6: Why Agile Planning Works