R for beginners: Some basic graphics code to produce informative graphs, part two, working with big data

A tutorial by D. M. Wiig

In part one of this tutorial I discussed the use of R code to produce 3d scatterplots. This is a useful way to produce visual results of multi- variate linear regression models. While visual displays using scatterplots is a useful tool when using most datasets it becomes much more of a challenge when analyzing big data. These types of databases can contain tens of thousands or even millions of cases and hundreds of variables.

Working with these types of data sets involves a number of challenges. If a researcher is interested in using visual presentations such as scatterplots this can be a daunting task. I will start by discussing how scatterplots can be used to provide meaningful visual representation of the relationship between two variables in a simple bivariate model.

To start I will construct a theoretical data set that consists of ten thousand x and y pairs of observations. One method that can be used to accomplish this is to use the R rnorm() function to generate a set of random integers with a specified mean and standard deviation. I will use this function to generate both the x and y variable.

Before starting this tutorial make sure that R is running and that the datasets, LSD, and stats packages have been installed. Use the following code to generate the x and y values such that the mean of x= 10 with a standard deviation of 7, and the mean of y=7 with a standard deviation of 3:

##############################################
## make sure package LSD is loaded
##
library(LSD)
x <- rnorm(50000, mean=10, sd=15)   # # generates x values #stores results in variable x
y <- rnorm(50000, mean=7, sd=3)    ## generates y values #stores results in variable y
####################################################

Now the scatterplot can be created using the code:

##############################################
## plot randomly generated x and y values
##
plot(x,y, main=”Scatterplot of 50,000 points”)
####################################################

As can be seen the resulting plot is mostly a mass of black with relatively few individual x and y points shown other than the outliers.  We can do a quick histogram on the x values and the y values to check the normality of the resulting distribution. This shown in the code below:
####################################################
## show histogram of x and y distribution
####################################################
hist(x)   ## histogram for x mean=10; sd=15; n=50,000
##
hist(y)   ## histogram for y mean=7; sd=3; n-50,000
####################################################

The histogram shows a normal distribution for both variables. As is expected, in the x vs. y scatterplot the center mass of points is located at the x = 10; y=7 coordinate of the graph as this coordinate contains the mean of each distribution. A more meaningful scatterplot of the dataset can be generated using a the R functions smoothScatter() and heatscatter(). The smoothScatter() function is located in the graphics package and the heatscatter() function is located in the LSD package.

The smoothScatter() function creates a smoothed color density representation of a scatterplot. This allows for a better visual representation of the density of individual values for the x and y pairs. To use the smoothScatter() function with the large dataset created above use the following code:

##############################################
## use smoothScatter function to visualize the scatterplot of #50,000 x ## and y values
## the x and y values should still be in the workspace as #created  above with the rnorm() function
##
smoothScatter(x, y, main = “Smoothed Color Density Representation of 50,000 (x,y) Coordinates”)
##
####################################################

The resulting plot shows several bands of density surrounding the coordinates x=10, y=7 which are the means of the two distributions rather than an indistinguishable mass of dark points.

Similar results can be obtained using the heatscatter() function. This function produces a similar visual based on densities that are represented as color bands. As indicated above, the LSD package should be installed and loaded to access the heatscatter() function. The resulting code is:

##############################################
## produce a heatscatter plot of x and y
##
library(LSD)
heatscatter(x,y, main=”Heat Color Density Representation of 50,000 (x, y) Coordinates”) ## function heatscatter() with #n=50,000
####################################################

In comparing this plot with the smoothScatter() plot one can more clearly see the distinctive density bands surrounding the coordinates x=10, y=7. You may also notice depending on the computer you are using that there is a noticeably longer processing time required to produce the heatscatter() plot.

This tutorial has hopefully provided some useful information relative to visual displays of large data sets. In the next segment I will discuss how these techniques can be used on a live database containing millions of cases.

To leave a comment for the author, please follow the link and comment on their blog: r – R Statistics and Programming.

---

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more…

from R-bloggers https://www.r-bloggers.com/r-for-beginners-basic-graphics-code-to-produce-informative-graphs-part-two-working-with-big-data/

If you’ve made resolutions, you don’t want to forget about them when the novelty of the new year has worn off. Nagbot is a fun texting app that helps you stick to your goals by texting you regular reminders.

To create a nag, you enter your name, phone number, and goal into Nagbot’s website. From there, you can choose how often and when you want Nagbot to nag you. You’ll also decide how mean you want it to be, from “You’ll get ‘em next time!” to “You’re dead to me.” The tool is similar to apps like Streaks in that it sends you daily (or weekly) reminders to work on a task. However, it’s a little sassier and simpler, and perhaps more importantly, it’s free. You can also opt out of the texts whenever you want with “STOP.”

Nagbot has a really simple goal tracking function, too. When it texts you, you get a link you can use to check your goal progress on Nagbot’s website. You obviously have to provide your number, but their privacy policy explicitly states, “Nagbot will not rent or sell potentially personally-identifying and personally-identifying information to anyone.”

Head to the link below to give it a try.

Nagbot via ProductHunt

from Lifehacker, tips and downloads for getting things done http://lifehacker.com/nagbot-sends-mean-texts-to-help-you-stick-to-your-resol-1791188092

from SaaStr http://www.saastr.com/why-the-law-of-large-numbers-is-just-an-excuse/

from Co.Labs https://www.fastcompany.com/3067232/what-happened-when-i-stopped-saying-sorry-at-work-for-a-week?partner=rss

from Stories by Caio Braga on Medium https://uxdesign.cc/design-automation-an-interview-with-matt-cooper-wright-from-ideo-edf6bb8547df?source=rss-27353c0f540b——2

A well-designed site isn’t how easy it is to use or how elegant it looks. A site isn’t well-designed unless the user is satisfied with their experience. An overlooked aspect of this experience is performance. A slow, beautiful site will always be less satisfying to use than an inelegant fast site. It takes a user just three seconds to decide to abandon a website.

“To the typical user, speed doesn’t only mean performance. Users’ perception of your site’s speed is heavily influenced by their overall experience, including how efficiently they can get what they want out of your site and how responsive your site feels.” – Roma Shah, User Experience Researcher

“A slow, beautiful site will always be less satisfying to use than an inelegant fast site.”

At the surface, performance is achieved through compression, cutting out extra lines of code and more, but there are limits to what can be achieved at a technological level. Designers need to consider the perceived performance of an experience to make it feel fast.

“There are two kinds of time: clock time and brain time.”

There are two kinds of time: clock time and brain time. The former is the objective measure of time; the latter is how a person perceives time. This is important to people involved in human-computer interaction, because we can manipulate a person’s perception of time. In our industry, this manipulation is called the perception of performance.

How Quick is Appropriate?

This visual demonstrates how we perceive time. Anything less than one second is perceived as ‘instant’ behaviour, it is almost unnoticeable. Up to one second is immediate, anything more than this is when the user realises they are waiting.

Instant behaviour could be an interface providing feedback. The user should not have to wait for this, they should get a message within 0.2s of clicking a button.

Immediate behaviour could be a page loading. The user should not have to wait any more than 1 or 2 seconds for the results they want to load.

If an interface needs that extra time, we should say ‘this may take a few more seconds’ and provide feedback on how long it will take. Don’t leave the user asking too many questions.

Active & Passive Modes

Humans do not like waiting. We need to consider the different modes a person is in when using a website or application: the active and passive modes. During the active mode users do not realise they are waiting at all; during the passive mode their brain activity drops and they get bored.

“It takes a user just three seconds to decide to abandon a website.”

You can keep people in the active mode by pre-loading content. Modern browsers do this while you are typing in a URL or searching in the address bar. Instagram achieves this by beginning to upload photographs in the background the moment you choose a photograph and starting creating the post to make the upload feel instant.

Instagram also shows an obscured preview of images that have not yet loaded.

“As designers, we should do everything we can to keep our users in the active mode.”

Display content as soon as you can to reduce the amount of time a user is in the passive mode. YouTube does this by streaming the video to the user despite it not being 100% downloaded. Instead, it estimates how fast the user can stream, and waits for that portion of the video to load, automatically chooses a bitrate, and starts playing it. Only buffering when absolutely necessary.

Both methods require us to prioritise the content we want, and load the rest of the page around it.

“Your page needs to load 20% faster for your users to notice any difference.”

Your page needs to load 20% faster for your users to notice any difference. If your page takes 8s to load today, a new version needs to take 6.4s to load for it to feel faster. Anything less than 20% is difficult to justify.

Helping Developers

Even if you understand all aspects of page speed, you should be thinking about it the moment you start creating a design system for a UI, working with the development team to fine-tune performance and figure out where marginal gains can be had.

This could be as simple as ensuring you provide loading states and fallbacks (failed states) to your developers so the user doesn’t have to wait for the entire page to load before they can read anything.

Here’s a short step-by-step guide to ensuring you are considering performance when designing:

• Research the priority content that should load in your interface. If it’s a news article, the text content should load first, allowing the user to start reading before the experience has even finished loading.
• Provide a loading state (e.g. placeholder content) and a fallback (e.g. un-styled text) for all elements you design and use.
• Work with the developers to fine tune performance and work out what technologies can be used to ensure quick loading (e.g. browser caching and progressive jpegs).

Slack take a common approach of placeholder content to imply what the user is going to see, making progress feel faster than it is. A blank screen here would be frustrating.

These tasks may seem complete, but it is important to revisit your work and fine-tune to make as many marginal gains as possible.

Measuring Performance

A way to measure perceived performance is by inviting users to navigate your site and asking them to estimate how long it took to load. Another option is to provide multiple experiences and ask which is faster.

A survey can be as simple as a scale like this one. Get enough answers, and you have a clear average.

The sample should be large enough to gather a realistic average that takes into account different perceptions and, if remote, the varying connection speeds of your participants.

“A site isn’t well-designed unless the user is satisfied with their experience.”

Once you have measured the perceived performance, you should continue to tweak it, perform research, and make further improvements. Things can only get better. Keep tweaking until it’s at a point you’re happy with it, then tweak again.

Further Reading

If you want to dig deeper into the perception of speed, check out the following resources:

from Sidebar http://sidebar.io/out?url=https%3A%2F%2Fblog.marvelapp.com%2Fa-designers-guide-to-perceived-performance%2F

Fraugster, a German and Israeli startup that has developed Artificial Intelligence (AI) technology to help eliminate payment fraud, has raised $5 million in funding.

Earlybird led the round, alongside existing investors Speedinvest, Seedcamp and an unnamed large Swiss family office. The new capital will be used to add to Fraugster’s headcount as it expands internationally.

Founded in 2014 by Max Laemmle, who previously co-founded payment gateway company Better Payment, and Chen Zamir, who I’m told has spent more than a decade in different analytics and risk management roles including five years at PayPal, Fraugster says it’s already handling almost $15 billion in transaction volume for “several thousand” international merchants and payment service providers, including (and most notably) Visa.

Its AI-powered fraud detection technology learns from each transaction in real-time and claims to be able to anticipate fraudulent attacks even before they happen. The result is that Fraugster can reduce fraud by 70 per cent while increasing conversion rates by as much as 35 per cent. The point of any fraud detection technology, AI-driven or otherwise, is to stop fraudulent transactions whilst eliminating false positives.

“We founded Fraugster because the entire payment risk market is based on outdated technology,” the startup’s CEO and co-founder Max Laemmle tells me. “Existing rule-based systems as well as classical machine learning solutions are expensive and too slow to adapt to new fraud patterns in real-time. We have invented a self-learning algorithm that mimics the thought process of a human analyst, but with the scalability of a machine, and gives decisions in as little as 15 milliseconds”.

Once integrated, Fraugster starts collecting transaction data points such as name, email address, and billing and shipping address. This is then enriched with around 2,000 extra data points, such as an IP latency check to measure the real distance from the user, IP connection type, distance between key strokes, and email name match. Then the enriched dataset is sent to the AI engine for analysis.

“At the heart of our AI engine is a very powerful algorithm which can mimic the thought process of a human analyst reviewing a transaction. As a result, we can analyze the story behind every transaction and say with precision which transactions are fraud and which aren’t,” explains Laemmle.

“You get a score or decision. Results are completely transparent (and not a black box), so you can understand exactly why a transaction was blocked or accepted. On top of this, our speeds are as low as 15ms. The reason why we’re so fast is because we’ve invented our own in-memory database technology”.

Fraugster cites competitors as incumbent enterprise level companies like FICO or SAS, which, it claims are based on outdated technology.

Adds Laemmle: “At Fraugster, we do not use any rules, models or pre-defined segments. We don’t use a single fixed algorithm to analyze transactions either. Our engine reinvents itself with every new transaction. This lets us understand transactions individually and therefore decide which one is fraudulent and which one isn’t. As a result, we can offer unprecedented accuracy and the ability to foresee fraudulent transactions before they happen”.

from TechCrunch https://techcrunch.com/2017/01/16/fraugster/?ncid=rss