The Volume, Velocity, Variety, and Visualization of Big Data

What if We had More Big Data Artists in the World?

Today’s Big Data Scientists are being challenged with discovering actionable insights from the Volume, Velocity and Variety of data resources in cost-effective innovate ways. This is the foundation of Big Data Trends and has tremendous value when understanding is Visualized by today’s Big Data Artists.

Big Data Artists and Data Storytelling

Data comes to life in the hands of Data Artists.  Effective Data Storytelling can inspire new ideas and actions. Statistics guru Hans Rosling is a modern day Michelangelo of Big Data.  Watch his TED Talk video about “global trends in health and economics“.  Hans really gets rolling into this data story at about the 4 minute mark of this video.

Big Data Virtualization

Processing unstructured, semi-structured, and structured data can be accomplished with open-source tools such as Hadoop, MongoDB, Node.JS, with multiple programming languages including Java and Python. Many of these Big Data Tools can be installed and configured within a few minutes.  We can also leverage Cloud Computing for processing Big Data. Cost-effective processing of Graph Data for Social Network Analytics can be accomplished with GraphChi.  My friend and colleague, Brad Cox, created an easy to use plugin to accelerate graph data processing.  There are a plethora of tools and resources out there.  Help others and share your favorites here.

Big Data Visualization

Getting the right information to the right person at the right time in the right way is a big deal.  Data Visualizations should not be cryptic like a puzzle, but easy to understand.  This is not a simple task and is the reason why we need more Big Data Artists in the world. My friend Angela Dunn has inspiring Data Visualization stories at Data is Beautiful.

Data Visualization Tools

Great artists experiment with a variety of material and tools to create their masterpieces.  The modern day Big Data Artist can experiment with these tools to create their next masterpiece.

Gephi The Open Graph Viz Platform: Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs.  This tool makes it fun to experiment with GraphData.

RapidMiner Data Mining System: RapidMiner is an open-source system for data mining. It is available as a stand-alone application for data analysis and as a data mining engine for the integration into custom products.

Sencha Ext JS is an “Enterprise grade JavaScript Framework”.  Ext JS Plugin-free Charting provides advanced charting and graphing capabilities, delivering pixel perfect visuals on any browser on any operating system.

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.

Highcharts is a charting library written in pure JavaScript, offering intuitive, interactive charts to your web site or web application. Highcharts currently supports line, spline, area, areaspline, column, bar, pie, scatter, angular gauges, arearange, areasplinerange, columnrange and polar chart types.

Raphaël is a small JavaScript library that should simplify your work with vector graphics.  Raphaël ['ræfeɪəl] uses the SVG W3C Recommendation and VML as a base for creating graphics. This means every graphical object you create is also a DOM object, so you can attach JavaScript event handlers or modify them later. Raphaël’s goal is to provide an adapter that will make drawing vector art compatible cross-browser and easy.

Visual.ly supports Visual Storytelling with data visualizations and infographics.     Visually’s Visual Reports can turn monthly performance metrics into a powerful visual narrative that communicates company objectives and milestones to both internal and external stakeholders. Infographics can also communicate product features.

Visualizing data helps others understand the value of information.  Additional data visualization tools are available here Top 20 Data Visualization Tools.  Visual content inspires conversation, as I was writing this, my friend Kathy Meyer shared this article about using visual content to inspire conversation 26 Ways to Use Visuals in Your Social Media Marketing

Big Data & Visualization Resources

Learning more about Big Data and the art of data visualization is a valuable skill to master.  There are several online classes and meetups in the area.  A recent meetup in the Washington DC area was Uncovering Hidden Social Information with Prof. Jennifer Golbeck. Her book, “Analyzing the Social Web” is available on Amazon.  Another great book to help you master the art of Big Data and Social Network Analytics is, “Mining the Social Web: Analyzing Data from Facebook, Twitter, LinkedIn, and Other Social Media Sites”.

Summary

Big Data is the Gold Rush of the 21st Century where discovering nuggets of actionable insights is transforming the way we work, live, & play.  Big Data Is Changing the Whole Equation for Business and organizations that fail to define Big Data Leadership will fall behind, far behind.  Organizations that hope to remain competitive in today’s market will include a strategy that implements Big Data tools along with the training and development of Data Scientists and Big Data Artists.

Collection of Social Business Predictions & Tech Trends for 2012

The Future of Social Business & Enterprise 2.0

Many Things in Motion

The Collective Intelligence of Social Business

We are about to embark on an innovative journey that will transform the workforce.  This new innovation age will blur the lines of work and play.  Integrating internal and external Social Media into Enterprise workflows to support collaborative business intelligence will become more of a priority.  The future of Social Business will empower the workforce, partners, and customers in the process of co-creation that will drive new levels of innovation across multiple markets.  These new business models will begin to harness the power of Collective Intelligence as the Enterprise explores collaboration beyond it’s main silo, the firewall.

“The whole is greater than the sum of its parts.” – Aristotle

The State of Social Business

Jeremiah Owyang’s “State of Social Business” presentation at LeWeb2011 covers why it is important to have a Social Business Strategy in this video.

Social Business Predictions & Trends for 2012

This year will set the stage for the decade of”radical innovation”.  Everyone may not have flying cars in 2022, but business models as we know them today will seem like ancient cave etchings.  There were several interesting predictions and trends in our Collection of Enterprise 2.0 Predictions & Tech Trends for 2011.  This will be the year of the Social Scientist and Data Artist.

  • Cloud Computing: Implementing cloud computing in the Enterprise and Government will be a prime focus.  This effort will start to mature beyond private virtual clouds and Software as a Service (SaaS) to include Platform as a Service (PaaS), Infrastructure as a Service (IaaS) and exploring Business Process as a Service (BPaaS).
  • Web-Oriented Architecture (WOA): web-oriented architecture extends service-oriented architecture (SOA).  It’s RESTful methods will help support cloud computing across multiple environments and devices.
  • Gamification: This is not angry birds in the workplace, but the use of Game Design Thinking for supporting collaboration and innovation.  Social Scientists will help organizations put an emphasis on psychology over technology to help others achieve the state of flow in business processes.
  • Mobile: Organizations will make real investments into supporting mobile technologies. Tablets and Smart Phones are slowly replacing the standard computers in the workforce.  The use of web-based apps and widgets on mobile devices will become more standard.
  • Big Data: Context will become the new king as organizations leverage internal and external data sources and social networks for business intelligence.
  • Enterprise App Stores & Business Intelligence Dashboards: Understanding information at the speed of business will help ensure success.  Visualizing business intelligence will move from IT departments to business departments.  Technologies like JackBe’s Presto Platform empower business leaders with the ability to “assemble” apps like lego blocks to support business decisions.
  • Identity Management:  Yes, the silos are crumbling and this includes corporate firewalls.  Supporting collaboration and secure communication across multiple environments and devices will have thought leaders thinking beyond firewalls and the use of Secure Socket Layers (SSL).  This shift from “need to know” to “need to share” will drive the use of intelligent point-to-point controls.  Attribute-Based Access Controls (ABAC) and Policy-Based Access Controls with eXtensible Access Control Markup Language (XACML) will be considered as alternatives to Role-Based Controls.  Additional thoughts about Identity Entitlement and Access Management (IdEA) are available at Jericho Forum.

Social Business Thought Leaders on Twitter

This Social Business Thought Leader twitter game board represents people focused on sharing ideas about Social Business Strategy.  Connect with more social business thought leaders on this twitter list: Social Business Strategy Leaders.

[twitterusers @rwang0 @oscarberg @dhinchcliffe @rhappe @rossdawson @krcraft @byAgah @raesmaa @absolutesubzero ]

A Collection of Social Business Predictions & Tech Trends

Update 01/01/2012

Social Intranets in Social Business

Collective intelligence structures that harness network effects improve as more people use them.  Learn more from “Social Intranets in Social Business” by Lee Bryant.

This new innovation age in Social Business will remind us of the Industrial Age, but instead of mining for precious gems of gold and silver, we will be mining for precious gems of knowledge.  Instead of factories that pollute our environment will have idea factories that improve our environments.

Enterprise 2.0 App Stores: When Good Web 2.0 Apps Go Bad

Custom Dashboards in the Enterprise & Web 2.0 Apps

There’s an app for that!

The success of Apple’s iPhone App Store, Mac apps, and Google’s Marketplace all play a part in driving the trend of Enterprise 2.0 App Stores in business organizations of all sizes.  The idea of providing a solution with “There’s an app for that!” will be common place in the near future.  The App Store market will get very interesting when organizations and Government Agencies harness the true power of Service Oriented Architecture (SOA) & Cloud Computing.  This trend will help fuel the Federated System.  More information about Enterprise 2.0 App Store Architecture can be found here The 80-20 Rule for Web 2.0 Architecture in the Enterprise.

Where Do Apps Come From?

Custom Enterprise 2.0 Dashboards can include apps, widgets, and gadgets that include resources that are internal, external, and a combination of both.

  1. Internal Resources: Apps and their data that are hosted and maintained within the organization. The risk level is low.
  2. Internal and External Resources: There are usually internally created apps that use external data. The risk level is medium.
  3. External Resources: Apps that are hosted by third parties.  The trust relationship is complex and the risk level usually remains high.

How Are Apps Delivered?

Apps are added to devices and dashboards in multiple ways. App code and private data should reside in the client, but this is rarely the case.  Web 2.0 Apps are usually added to Enterprise 2.0 Dashboards by using the following technologies and methods.

What Are Application Security Risks?

Attackers can potentially use many different paths through your application to do harm to your business or organization. Each of these paths represents a risk that may, or may not, be serious enough to warrant attention.  The top 10 application security risks of 2010 can be reviewed on the Open Web Application Security Project (OWASP) web site here.  Additional Web Security information is available from the Open Ajax alliance at Ajax and Mashup Security.  The main rule of thumb is, “Never trust external data”.  Using a Proxy Server to fetch external data can help support the additional security requirements.  A proxy server is also helpful in capturing metrics of external resource usage.  The proxy server can integrated like an Enterprise Service Bus (ESB) to support the complex structure of Enterprise 2.0 App Stores.

What Happens When Good Web 2.0 Apps Go Bad?

How to Detect a Key Logger on Your System

Most people in the Web 2.0 World are familiar with the acronym WYSIWYG, “What You See is What You Get”.  This new acronym WYRIWYR, “What You Requested is What You Received” will be covered here.  The consumer and the producer should be focused on WYRIWYR.  Producers need to trust the consumer’s identity and consumers need to feel secure.

Data can be tampered with on either end and while in transit.

The Open Source Software Community frequently uses checksum to protect software integrity. This same strategy can be used to protect consumers from malicious apps and widgets.  This simplified example will use MD5 in PHP to check the integerity of the app, but MD5 should not be used for sensitive data like passwords in a production environment.  US-CERT of the U. S. Department of Homeland Security said MD5 “should be considered cryptographically broken and unsuitable for further use,” and most U.S. government applications will be required to move to the SHA-2 family of hash functions after 2010.

Simple App

Here is a very simple app that could be part of a custom Enterprise 2.0 Dashboard. The App is reviewed and approved. The reviewer signs the app (creates app MD5 Hash: c15a7308d89afe9218a1b0f60a37f8ad) so changes can be detected when it comes back through the proxy server.





World's Best Web 2.0 Widget

Simple App in Proxy Server before Dashboard Display
Deliver app if new hash and signature match. Disable app and notify Admin if something does not look right.

$app = <<



World's Best Web 2.0 Widget

END; if (md5($app) == 'c15a7308d89afe9218a1b0f60a37f8ad'){ deliverApp($app); //Signature & Hash matched! App Delivered. } else{ disableApp($appId); alertAdmin($message); }

The Simple App with Key Logger Script Injected

$app = <<




World's Best Web 2.0 Widget

END; if (md5($app) == 'c15a7308d89afe9218a1b0f60a37f8ad'){ deliverApp($app); } else{ disableApp($appId); alertAdmin($message); //Signature & Hash do not match App not delivered. //App disabled and Admin notified. }

Happy Fav Five Friday!

Fav 5 Places

  1. Google Gadgets For Your Webpage
  2. ‘App store’ makes service orientation real for the business
  3. Global CIO: The Case For Copying Apple’s App Store
  4. Nexuo Enterprise Platform
  5. Enterprise Irregulars: Designing User Experience

The people from Open Social provides a great Introduction To Signed Requests

OpenSocial API provides a method to communicate OpenSocial ID numbers back to your server in a secure way, allowing for the construction of robust web service backed OpenSocial applications, using a portion of the OAuth authorization protocol.  This article will explain the method to make such secure requests from your OpenSocial applications, as well as the server-side process that you need to follow in order to verify that the data passed has not been tampered with.   Learn more here.

The 80-20 Rule for Web 2.0 Architecture in the Enterprise

Enterprise 2.0 Cloud Computing & Service Oriented Architecture (SOA)

Integrating Internet Principals In Your Intranet

80-20-Service-Oriented-Architecture-SOA

You may of heard someone in your office say something like “20% of the people are doing 80% of the work”.  This is known as The Pareto Principle, aka the 80-20 Rule.  This principal will also apply to the future architecture of the internet composed of 80% Consumers and 20% Producers supported by “Linked Data”  This paradigm shift in information architecture will be fuled by the adoption of using widgets, gadgets, and apps to connect information.

Enterprise 2.0 App Stores

  • What if you had an Enterprise 2.0 App created from SharePoint?
  • What if you had an Enterprise 2.0 App for email?
  • What if you had Enterprise 2.0 Apps for your team blog, wiki, graphs, activity streams, …?
  • What if all these apps were displayed as Widgets in your Enterprise 2.0 Dashboard?

Why would you log into all these different areas, when you can see what you want, how you want in a single place.  Navigating to multiple places is time consuming, even with Single Sign On (SSO) solutions in place.  Most Enterprise 2.0 Apps, Widgets, and Gadgets can be created with common web programming languages.  These web programming languages can be as simple as standard HTML, CSS, and JavaScript.  These User Interfaces can easily leverage AJAX and other Web 2.0 Technologies to enhance the User Experience.  The extra benefits of using these standard technologies is lower development cost, shorter time to deployment, and better support across multiple devices such as smart phones, iPads, and standard tablets.

Enterprise 2.0 App Store Architecture

Most organizations have a difficult time measuring the use of their Information Technology.  Some organizations can measure enough to know that moving at least some technology to the cloud makes sense.  Businesses can also realize the benefits from the concept of cloud computing through Enterprise 2.0 App Store Architecture.

What is an Enterprise 2.0 App?

An Enterprise 2.0 App is “a snippet of code that is shared to serve a specific purpose”.  This “App” can be displayed and used on smart phones, mobile devices, desktops, web browsers, dashboards, or other applications.

Where is the App Stored?

Apps can be stored internally or externally.

Internal Apps, Widgets, & Gadgets

Internal apps are hosted and used by the owner, usually in the same domain, therefore the level of trust is higher.

External Apps, Widgets, & Gadgets

External apps are hosted outside the domain and usually by a third party.  The level of trust is lower, therefore the proper security measures should taken.  All developers should know how to build, design and test the security of web applications and web services.

Using a Web Proxy

All modern web browsers impose a security restriction on network connections, which includes calls to XMLHttpRequest. This restriction prevents a script or application from making a connection to any web server other than the one the web page originally came from (Internet Explorer will allow cross-domain requests if the option has been enabled in the preferences). If both your web application and the XML data that application uses come directly from the same server, then you do not run into this restriction.  See the PHP Web Proxy code below and learn more about AJAX Web Proxies here.

<?php
// PHP Proxy example for Yahoo! Web services.
// Responds to both HTTP GET and POST requests
//
// Author: Jason Levitt
// December 7th, 2005
//

// Allowed hostname (api.local and api.travel are also possible here)
define (‘HOSTNAME’, ‘http://search.yahooapis.com/’);

// Get the REST call path from the AJAX application
// Is it a POST or a GET?
$path = ($_POST['yws_path']) ? $_POST['yws_path'] : $_GET['yws_path'];
$url = HOSTNAME.$path;

// Open the Curl session
$session = curl_init($url);

// If it’s a POST, put the POST data in the body
if ($_POST['yws_path']) {
$postvars = ”;
while ($element = current($_POST)) {
$postvars .= urlencode(key($_POST)).’=’.urlencode($element).’&';
next($_POST);
}
curl_setopt ($session, CURLOPT_POST, true);
curl_setopt ($session, CURLOPT_POSTFIELDS, $postvars);
}

// Don’t return HTTP headers. Do return the contents of the call
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);

// Make the call
$xml = curl_exec($session);

// The web service returns XML. Set the Content-Type appropriately
header(“Content-Type: text/xml”);

echo $xml;
curl_close($session);

?>

You should also see this article: “Restricting Access to your AJAX Services“.

Open Source to the Rescue!

Open Source Libraries and Frameworks to Support Enterprise 2.0  Apps, Widgets, & Gadgets

Creating Enterprise 2.0 Apps, Widgets, and Gadgets can be less time consuming with these free open source libraries and frameworks.

jMaki

jMaki is a lightweight client/server framework for creating JavaScript centric Web 2.0 applications using CSS layouts, widgets widget model, client services such as publish/subscribe events to tie widgets together, JavaScript action handlers, and a generic proxy to interact with external RESTful web services. While jMaki abstracts much of the JavaScript and CSS by providing defaults for widgets, the JavaScript widgets and CSS are made easily accessible so they may be customized by a designer or page developer. jMaki focuses on the aspects of delivering JavaScript to the client allowing the JavaScript to communicate to various server-technologies including PHP, Java (JSP/JSF), and Phobos in a server-technology neutral way. Learn more here.

jmaki-architecture

Google Web Toolkit (GWT)

Google Web Toolkit (GWT) is a development toolkit for building and optimizing complex browser-based applications. Its goal is to enable productive development of high-performance web applications without the developer having to be an expert in browser quirks, XMLHttpRequest, and JavaScript. GWT is used by many products at Google, including Google Wave and the new version of AdWords. It’s open source, completely free, and used by thousands of developers around the world. Learn more here.

Screen shot 2011-01-14 at 2.52.57 AM