The problem with randomness

How to generate random numbers from spam

Random number generators: The devil is in the details

I found SecurityDump’s WPRandom the other day:

Generating random numbers is pretty complicated if you need them for cryptographic algorithms. This software generates them based on spam comments…

It caught my eye as a sort of “spinning spam into RNG gold”, or more likely, PRNG (pseudo-random number generated) gold. Many WordPress blogs, whether self-hosted using WordPress.org or not, effectively use Akismet as a comment spam sieve.

SHODAN related infosec assortment

The other Defcon

I never attended DEFCON, though it remains a dream I hope to realize one day, soon. It may soon become too logistically awkward due to increasing numbers of attendees.

Shodan is a remarkable search engine. Traditional search engines use “spiders” to crawl websites. Shodan culls data from ports. It was created by John Matherly in 2007. He continues to develop it.

Shodan is helpful for locating web server vulnerabilities. It is available as a free service, for up to 50 searches. Query syntax includes searches by country, host name, operating system and port. Shodan can search for software AND hardware. It has been acknowledged by mainstream media. The most prominent coverage was in early June, via The Washington Post, when Stuxnet received so much press attention.

Me and Shodan

Next is my Scribd infosec collection. It isn’t exclusively Shodan-related. This is why.

Zanran is a new data search engine

Something new and different in search has appeared.

Zanran is an internet start-up company that hails from somewhere other than Mountain View or Sunnyvale, California. Nor is it in “Silicon Valley East”, the new incubator of technology ventures otherwise known as the Borough of Manhattan. Zanran is farther than farthest Fishkill, across a span greater than even the Tappan-Zee can bridge. Zanran is a U.K. domiciled company in Islington, London.

Not a Google Universal Search 2.0 competitor

Zanran seems to be more of a database searching tool. It would probably be best classified as a specialized search engine.

Zanran Search beta version: screen shot

Zanran’s search method is described as patented but based on open-source programs. The actual patent, which I only glanced at, A Method and System of Indexing Historical Data, should help in clarifying. Zanran distinguishes itself because it is particularly well-suited to web search for information that has embedded numerical or graphical data:

Zanran helps you to find ‘semi-structured’ data on the web… numerical data e.g. a graph in a PDF report, or a table in an Excel spreadsheet, or a bar chart shown as an image in an HTML page. This huge amount of information can be difficult to find using conventional search engines, which are focused primarily on finding text… Put more simply: Zanran is Google for data.

Zanran is not a search engine with obvious uses in text or sentiment analysis. The beta website has a long page of examples demonstrating the speed (fast), breadth (across a very diverse assortment of scientific and analytic use cases) and quality of results.

Arthur Weiss, a competitive analyst and former long-time employee of Dun & Bradstreet UK, did a very thorough review of Zanran Search (April 2011):

I’ve been playing with a new data search engine called Zanran… The site is in an early beta. Nevertheless my initial tests brought up material that would only have been found using an advanced search on Google – if you were lucky. As such, Zanran promises to be a great addition for advanced data searching.

Zanran enters the marketplace

Zanran appears to have retained Mallard Digital Marketing. Mallard Digital’s hallmarks are “Authenticity, Transparency and Engagement”. Mallard features an attractive duck in the company logo, and in this rather engaging 15-second video. I base my conjecture about Mallard and Zanran upon three pieces of evidence:

1. Mallard’s recent announcement, about the acquisition of a search engine as a new client on 29 March 2011
2. The Zanran company dog enjoyed playing with Mallard’s Labrador retriever in March 2011 (also via Facebook)

Analogy and Digression: SHODAN

As a very general analogy, Zanran functionality reminds me of SHODAN computer search. SHODAN is a search engine that can be used to:

find specific computers (routers, servers, etc.) … [it is] a search engine of banners. Google and Bing are great for finding websites. But what if you’re interested in finding computers running a certain piece of software (such as Apache)?  Maybe a new vulnerability came out and you want to see how many hosts it could infect?

Here’s a screen shot of the main query page:

SHODAN search engine: screen shot

I am impressed to no end with SHODAN. It is quite clever, and remains very low profile, much like my blog.

UPDATE

I drafted this on 12 May 2011 but failed to actually post due to my insatiable need to excessively fuss and play with WordPress functionality. In the interim, others (most notably Search Engine Journal) have also found the subject of the following post, the Zanran data search engine. I mention this not as self-promotion, but rather, to emphasize that Zanran may be of greater significance than my casual tone indicates.

Don’t Bring Your Guns to Town

From The New York Times, October 5, 2010:

Handgun permit holders who have recently seen their rights greatly expanded by a new law — one of the nation’s first — that allows them to carry loaded firearms into bars and restaurants that serve alcohol… Tennessee is one of four states, along with Arizona, Georgia and Virginia, that recently enacted laws explicitly allowing loaded guns in bars. Previously, states like Tennessee did not allow its residents to carry concealed weapons unless they had a special permit from the local authorities.

©2010 The New York Times All rights reserved. Used by permission and protected by the Copyright Laws of the United States. The printing, copying, redistribution, or retransmission of the Material without express written permission is prohibited.

I was troubled when Arizona recently passed legislation to remove the requirement to obtain a concealed weapons permit to, well, carry a concealed weapon on one’s person.  Carrying a loaded weapon in a bar is much worse. It is a profoundly bad idea. That is a blatant statement of opinion. However, I am a rifle marksmanship enthusiast and dearly miss my wood-stock, .22 Remington bolt-action long rifle that I sold when I went off to attend the Wharton School. I am not biased against the Second Amendment. Yet I do think this law shows a serious breach of common sense.

Minor mysteries of spam

Information overload has been one of my recent concerns. Spam certainly exacerbates the situation! I am spared the worst of spam, due to the minimal traffic on my websites, although I was treated to a glut of spam from around the world immediately after I posted that skateboard video a few months ago.

Spam versus content over time

Spam is diverse. It manifests as spam email, spam comments, spam blogs (known as splogs) and all-unoriginal websites of reposted content.

Spam deterrence

Akismet, an Automattic site, provides excellent, free of charge anti-spam services to WordPress.com blog sites such as mine. Akismet maintains a daily Stats Page including a graph of ham versus spam, for the past five years. Ham is Akismet’s term for a non-spam message. I was pleased to see this post Do you appreciate Akismet?

If so, please take a moment to leave a short comment on this post letting us know! We’re working on a new site design and would love to include some new testimonials.

Ironically, I was unable to complement Akismet, as commenting was disabled.

In addition to Akismet, WordPress suggests using a word list filter of one’s own. As a utility, WordPress will match your list against incoming comments. If any matches are found, that comment is flagged and immediately redirected into the spam bin, for the blog admin to review or just auto-delete. I’ve honed my word list for several months. I enjoy reading through the file every time I make any additions. It is such an odd and illicit list of words and phrases!

Use cases

I regret that I deleted all the skateboard video spam comments that slipped past both Akismet and my keyword filtering system, as they were the most spectacular of all. These are selected excerpts and replies from another blog’s spam comments:

“I see you own a good blog.”
Fool! I’ve seen my content!
“Thanks! This post helps me with a school assignment.”
Fool! You’re gonna fail that one!

A recent spammer tactic is to meaningfully respond to the content of a blog post but also link to a solely commercial and often tacky webpage:

One was an XML editor and the other was a video on “how to be a hacker” (of the LulzSec variety as opposed to kernel patcher type).

CAPTCHA Content

Peculiar CAPTCHA Content

Are CAPTCHA selections audience-maturity rated? Regardless of rating, this was especially odd: “them urethras”

.

Random Numbers in Java

This post is about alternatives to the java.util.Random class, the most commonly used method to generate random numbers in Java.

Random: Playing the odds in Ancient Rome Osteria della Via de Mercurio

 java.util.Random

This class generates “random enough” values, which are described as having an informal level of randomness. Such numbers have a superficial appearance of randomness to an observer, usually a human observer, not a machine. These are considered low-quality random numbers. They should never be used in any security application.

True randomly-generated numbers must have the following characteristics:

• Any number is equally likely to be generated on every iteration. This is also known as a uniformly distributed series of numbers.
• The second criterion follows directly: No dependency exists between successive numbers as they are generated.

Alternatives to better match your random needs

Neil Coffey of Javamex describes three alternative random number generation methods. Each approach continues to use class java.util.Random, while replacing the underlying algorithm. The first, called the XORShift generator, produces medium-quality random numbers very quickly, with only a single state variable and very simple code. This method is very well suited to J2ME games.

The next algorithm generates much higher-quality random numbers. It is a combined generator, using two XORShift generators of the sort described above. Mr. Coffey provides the code and explanation for the algorithm. This combined XORShift yields good-quality random numbers. It is suitable for non-gambling games and simulations, although it runs slightly slower than java.util.Random.

A cryptographic quality random number generator should have the following properties:

1. It should be impossible to predict prior and future numbers from any number generated;
2. The numbers should have no discernible biases;
3. The generator has a large period;
4. The generator can seed itself at any position within that period with equal probability.

A cryptographically secure random number generator is appropriate for security applications such as producing a web-server session id or picking encryption keys. Very high-quality random numbers are generated using java.security.SecureRandom as the replacement for java.util.Random. The trade-off in quality versus CPU cycle consumption is hardly surprising.  java.security.SecureRandom runs 20 to 30 times slower than any of the other algorithms.

Relative comparison between the methods

Here is a very simplified way of understanding the difference between each algorithm.

Let’s say that we need to generate a 128-bit encryption key. java.security.SecureRandom actually picks from a pool of 2 raised to the 127th power number of possible keys. Of course java.util.Random can also be used to generate a 128-bit key. However, the values will be selected from a smaller pool of numbers, on the order of 2 raised to the 47th power number of possible keys. This is because java.util.Random has a much shorter period, equal to 2 raised to the 48th power.

The single XORShift generator method falls between the two, as it has a slightly longer period, of 2 raised to the 64th power. The combined XORShift generator approach extends the period a bit further.

Note than neither java.util.Random nor either of the XORShift generators are seeded randomly. This is why java.security.SecureRandom, with a machine-generated and much more truly random random seed, is superior.

* The machine-generated random seed is what is called entropy in random number generators.

Social Web Pathology Part 1

Share and Share A-like

Note: this is an updated version of my April 28th post on the theme of Social Web Pathology.

I’ve been pondering the theme of “Social Web 2.0 Pathology: Are We Connected Yet?”, and will introduce it with this mild example. Today’s post will then assess recent developments in our vanishing degrees of separation.

The date of this graphic was April 16, 2010 thus it does not contain the very consequence-laden “Like” button. Facebook announced the release of the Like button to the World Wide Web domains-at-large at the F8 conference on April 22.

There is a recent development, and it should encourage advocates of the individual’s right to privacy. U.S. Congressman Charles Schumer has an awareness of the potential for abuse via social networking, due to accelerated data sharing.  He seems to have focused his concerns around the announcement at F8 of Facebook’s Open Graph and Like button data collector. Media coverage by wire services such as Reuters – Schumer vs Facebook was prompt: Congressman Schumer explicitly stated his concern over negative impact on the consumer’s right to privacy. Shortly after, Schumer was joined by other lawmakers, who proposed a sweeping internet privacy bill on May 4, 2010, indicating grave concern over the potential risks due to information disclosure resulting from participation in social networks. Not surprisingly, Facebook was mentioned by name, as it has been on the leading edge of the social media innovation.

What abuses might result from the Like button, and by whom? The abusers could be hackers aka crackers. Their method of infiltration might include harvesting of newly accessible personal data (mostly due to recent trends in over-sharing) using social engineering techniques. Alternatively, the traditional exploits involving unauthorized elevation of privilege would remain viable. Such is now facilitated by easier password breaking due to a larger pool of availale data. This becomes more straightforward as we tie ourselves closer and tighter together, along with our most critical identifying information via Social Networking sites and applications.

The other source of abuse might be from Facebook itself, due to sharing of user’s personal information with third-parties. FB announced policy that users will no longer have an opt-out feature available, regarding disclosure of certain basic identifying data.

Maybe the need to prove to one and all that “I have the most friends”, a sort of Web 2.0 version of Thorstein Veblen’s concept of “status seeking” and “conspicuous consumption”, will be reined in (see Professor Veblen’s Theory of the Leisure Class, 1899). While the impulse is present in most of us, many resist the temptation. FB was well-designed, though, because it seems to have overwhelmed common sense for even the most reticent of us. Similar observations could be applied to the rise of Farmville. With 86 million players, this game that seems primarily motivated by guilt (Professor Howard Linn wrote a paper on the subject shortly before his death earlier this year) has spread like wildfire. We will hope that Congressional scrutiny will slow the onslaught of the Like button, and hold it in check until the consequences are more carefully considered. 

We Revoke Our Right to Privacy with Assist From Facebook

At the annual Facebook F8 event yesterday, Facebook announced several powerful applications to expand the reach of the Web 2.0’s social networking model. New product announcements included Open Graph application software and inclusion of location-based services for greater social web inter-connectivity. The debut of the highly anticipated “Like” button on sites external to Facebook was also disclosed. In fact, Facebook stated that it already has established partnerships with approximately 30 highly visible websites including Yelp and Microsoft. Senior Facebook developers successfully demonstrated some of the potential uses to marketing and business users as well as individuals.

In light of this event, I’d like to share some of my concerns regarding the effect of increased information disclosure on us. The legal and contextual basis for my concern is drawn from a prescient feature article via BBC News. The following excerpt regarding erosion of privacy due to online activity is solid background material. It is followed by my own comments, as motivated by yesterday’s F8 activities.

How Online Life Distorts Privacy Rights for All by Zoe Kleinman, BBC News, 8 Jan 2010

People who post intimate details about their lives on the internet undermine everybody else’s right to privacy, claims an academic. Dr Kieron O’Hara [a senior research fellow in Electronics and Computer Science at the University of Southampton] has called for people to be more aware of the impact on society of what they publish online. “If you look at privacy in law, one important concept is a reasonable expectation of privacy,” he said. “As more private lives are exported online, reasonable expectations are diminishing.”

The rise of social networking has blurred the boundaries of what can be considered private, he believes – making it less of a defense by law. We live in an era that he terms “intimacy 2.0″ – where people routinely share extremely personal information online. “When our reasonable expectations diminish, as they have, by necessity our legal protection diminishes.”

This is the article’s most critical point: “People who post intimate details about their lives on the internet undermine everybody else’s right to privacy… When our expectations diminish, by necessity our legal protection diminishes…”

The negative effects associated with such a loss of privacy are significant. Contemplate an analogy, between the social web’s escalating trend toward over-sharing and the risk scenario implicit in the herd immunity effect of inoculation to prevent infectious disease.

Start by considering our rapidly growing social web. Achieving today’s level of coverage and confidence in the security of our private information was not achieved easily nor quickly. Certain rights, such as protected health information laws under HIPAA, were only legislated during the past five or ten years. However, in this sudden outpouring of social web-motivated information disclosure, we might destroy all that our predecessors, and ourselves, have fought to achieve over a span of decades or centuries.

Now consider immunization against contagious disease. Note that metaphorical comparison to disease is not overstatement: In a worst case scenario, the threat to personal privacy or infrastructure security due to unprecedented levels of network connectivity is dire. We’ll use the non-controversial example of polio, for which immunity is achieved at the community level. Decades of time and effort were required to wipe out incidence of this disease. However, by choosing not to protect oneself (or one’s children), the entire community is made more vulnerable, not just those who choose not to immunize. The immuno status of the “herd” can be compromised by a small number of now unprotected individuals.

The phrase “going viral” with the spread of social web information is in fact just as sinister as the original epidemiological context from which it was derived.

That is why nearly everyone who works in information security—be it computers, telephony, healthcare, or in financial or IT audit— seems leery of Facebook services. Every time I scan news feeds from InfoSec Island, Wired.com, or DEFCON, I notice references to the latest Facebook scam, exploit, user vulnerability etc. The data security and information technology community appear to be of one mind about Facebook: no system safeguards, no matter how stringent, can protect users who choosing to divulge information by over sharing.

I conclude with a link to visual editor and journalist Robb Montgomery’s advice about how to guard your privacy and enjoy your time more securely on Facebook.

Tech Update One: The Real Smart Card Finally Steps Up, Finally Steps Up

Today I step out from under my dark cloud of foreboding to bring glad tidings! It seems that the consumer is finally able to avail herself of effective and affordable protection from identity theft and collateral loss. Mine arrived in a small securely wrapped parcel from www.PayPal.com a few weeks ago: an ICT Display Card. After a ten-year wait, this form of the long-anticipated “Smart Card” finally debuts.

What does it do, and is it really anything special? Yes, because the ICT Display card appears to offer the first instance of double password protection (dual factor authentication) for the average consumer’s online transactions. Let me describe the process, although I cannot fathom how it works. The account holder logs in to PayPal servers via secure https connection using her established account name and password. After gently depressing the small rubbery nub, the ICT Display Card generates a six to eight digit security key.

How does that new password protocol work?

   It appears on a (possibly LED) display, flush with the surface of the card, on the upper corner. The user then keys in the numeric code, no other process nor hardware needed. After a 6 to 10 second pause (the instructions are contrite, asking for the user’s patience during that nearly imperceptible interval), the key is authenticated and account access is granted. A different randomly generated security key is created for every session, according to the instructions. One could also use a security token delivered via a text message, instead of the card-based security key.  

This nifty little card is the size of an ATM or credit card. It is thinner and lighterthan most office building entrance card readers. The only cost associated is a one-time charge of $5.oo, including shipping and handling.  Remarkable technical innovation was required, as the card is powered by a super lightweight, paper-thin, very long-life battery, which emits a low-power radio frequency transmitting the security code. But where is the receiver? The card is not location-dependent, and may be used with any login, with any IP address. I am very curious how it works!

The developers are a privately-held company, with numerous overseas retail banking customers, and a very low profile website, probably due to this extremely valuable proprietary technology.  PayPal offered this option  for a more secure connection to customers as a bullet point update on the login screen, as opposed to a more visible email distribution to customers.  In fact, I recall seeing it announced only once, with minimal promotion. Instructions are given for users in the USA, Australia, Canada, Great Britain, Germany and Austria. though I believe that PayPal offers the double password option solely to US-domestic customers at present. Actually, I am intrigued by the lack of fanfare as much as the capability of the card itself!