Connect with us

Technology

Order From Chaos: How Big Data Will Change the World

Published

on

Order from Chaos- Big Data

Order From Chaos: How Big Data Will Change the World

Harnessing the exponential surge in data creates big opportunities.

Thanks to Purefunds Big Data ETF (BDAT) for helping us put this together.

IBM estimates that each day, 2.5 quintillion bytes of data are created or replicated. That’s the equivalent of a million hard drives filling up with data every hour.

The current volume of data created is substantial: it is so much that 90% of the world’s data has been created in the last two years. However, the amount of information today pales in comparison to what our future holds, as the rate at which data is created is accelerating exponentially.

It’s for this reason that The Economist estimates that there will be roughly 7x more data in 2020 than there was in 2014.

Where Does Big Data Come From?

Big data comes from both internal and external sources. Internally, millions of old documents and records are scanned and archived by businesses. Most of the time, no detailed analytics have ever been run on this information. Externally, the public web offers millions of data sets published for public consumption by government, economic, census, and other sources.

There’s also a broad spectrum of data that exists that can be a part of both of these categories: social media posts, documents, emails, business applications, machine log data, media, and sensor data can all be collected, processed, and analyzed. To get a sense of the extent of this information, here’s what is created every hour just from social media and email: 72 hours of video uploaded to Youtube, 4 million search inquiries on Google, 200 million emails sent, 2.5 million shares on Facebook, and 300,000 tweets made.

Big Data = Big Opportunities for Business

With proper analysis, Big Data can lead to new understandings of consumer behaviour, better management decisions, new innovations, and improved risk management. However, there are big challenges in making use of so much information.

  • Too much data creates an information overload.
  • Organizing and storing all of this data can be problematic.
  • Companies don’t know how to use all of this data to create insight.

To organize and make sense of it all, data scientists use the three V’s to describe Big Data.

Volume is the scale at which data is created, and includes the massive amounts of information derived from phones, internet users, machine logs, and internet of things.

Velocity is the analysis of streaming data: for example, modern cars have 100 sensors that monitor different systems in real-time.

Variety is the different forms of data, and it reflects the fact that data comes in all shapes and forms. Finding a way to harmonize multiple types of data can be quite a challenge. Research finds that organizations spend up to 80% of their time modelling and preparing data, rather than actually gaining insight.

Let’s see how companies have been able to use Big Data to create opportunity.

Case Studies of Big Data

Macy’s adjusts pricing in near-real time for 73 million items based on demand and inventory.

American Express developed predictive models that analyze historical transactions and 115 variables to forecast the loyalty of customers. Using this data, they can see if customers may be potentially closing their accounts in the near future. Launching a pilot program in Australia, the company can now identify 24% of accounts in the country that will close in the next four months.

Walmart built a new search engine for their website that includes semantic data relying on text analysis, machine learning, and even synonym mining to create better search results. Online shoppers have been more likely to complete purchases as a result by 10% to 15%, increasing revenue by billions.

Los Angeles and Santa Cruz police departments have used an algorithm that is typically used to predict earthquakes, now using it to look at crime data. The software can predict where crimes are likely to occur down to 500 square feet. In areas the software is being used, there has been a 33% reduction in burglaries and a 21% reduction in violent crimes.

Big Market

Today’s data centers occupy the land to equivalent to almost 6,000 football fields. By 2020, the amount of digital information is expected to increase exponentially to more than 7x of what it is today.

In healthcare alone, Big Data is expected to eventually save $300 billion per year in healthcare analytics. Retailers may increase margins up to 60% through Big Data analytics.

“Information is the oil of the 21st century, and analytics is the combusion engine.” – Peter Sondergaard, Gartner Research.

Subscribe to Visual Capitalist
Click for Comments

Technology

Visualizing the Top U.S. States for AI Jobs

Nearly 800,000 AI jobs were posted in the U.S. throughout 2022. View this graphic to see a breakdown by state.

Published

on

Visualizing the Top U.S. States for AI Jobs

Much ink has been spilled over fears that artificial intelligence (AI) will eliminate jobs in the economy. While some of those fears may be well-founded, red-hot interest in AI innovation is creating new jobs as well.

This graphic visualizes data from Lightcast, a labor market analytics firm, which shows how many AI-related jobs were posted in each state throughout 2022.

In total there were 795,624 AI jobs posted throughout the year, of which 469,925 (59%) were in the top 10. The full tally is included in the table below.

RankStateNumber of job postings% of total
1California142,15417.9%
2Texas66,6248.4%
3New York43,8995.5%
4Massachusetts34,6034.3%
5Virginia34,2214.3%
6Florida33,5854.2%
7Illinois31,5694.0%
8Washington31,2843.9%
9Georgia26,6203.3%
10Michigan25,3663.2%
11North Carolina23,8543.0%
12New Jersey23,4472.9%
13Colorado20,4212.6%
14Pennsylvania20,3972.6%
15Arizona19,5142.5%
16Ohio19,2082.4%
17Maryland16,7692.1%
18Minnesota11,8081.5%
19Tennessee11,1731.4%
20Missouri10,9901.4%
21Oregon10,8111.4%
22Washington, D.C.9,6061.2%
23Indiana9,2471.2%
24Connecticut8,9601.1%
25Wisconsin8,8791.1%
26Alabama7,8661.0%
27Kansas7,6831.0%
28Arkansas7,2470.9%
29Utah6,8850.9%
30Nevada6,8130.9%
31Idaho6,1090.8%
32Oklahoma5,7190.7%
33Iowa5,6700.7%
34South Carolina4,9280.6%
35Louisiana4,8060.6%
36Kentucky4,5360.6%
37Nebraska4,0320.5%
38Delaware3,5030.4%
39New Mexico3,3570.4%
40Rhode Island2,9650.4%
41New Hampshire2,7190.3%
42Hawaii2,5500.3%
43Mississippi2,5480.3%
44Maine2,2270.3%
45South Dakota2,1950.3%
46Vermont1,5710.2%
47North Dakota1,2270.2%
48Alaska9700.1%
49West Virginia8870.1%
50Montana8330.1%
51Wyoming7690.1%

The following chart adds some context to these numbers. It shows how the percentage of AI job postings in some of the top states has changed since 2010.

We can see that California quickly became the primary destination for AI jobs in the early 2010s, presumably as Silicon Valley companies began developing the technology.

California’s share has since declined, with a significant number of jobs seemingly moving to Texas. In fact, many tech companies are relocating to Texas to avoid California’s relatively higher taxes and cost of living.

The 10 Most In-Demand Specialized Skills

Lightcast also captured the top 10 specialized skills that were required for AI-related jobs. These are listed in the table below.

SkillFrequency (number of postings)Frequency (% of postings)
Python296,66237%
Computer Science260,33333%
SQL185,80723%
Data Analysis159,80120%
Data Science157,85520%
Amazon Web Services155,61519%
Agile Methodology152,96519%
Automation138,79117%
Java133,85617%
Software Engineering133,28617%

If you’re interested in a career that focuses on AI, becoming proficient in Python is likely to be a good first step.

Continue Reading

Subscribe

Popular