Surface Web Deep Web & Dark Web

The Articles From Wikipedia, the free encyclopedia:

The Surface Web (also called the Visible Web, Indexed Web, Indexable Web or Lightnet)^[1] is the portion of the World Wide Web that is readily available to the general public and searchable with standard web search engines. It is the opposite of the deep web.

According to one source, as of June 14, 2015, Google's index of the surface web contains about 14.5 billion pages.

The deep web,^[1] invisible web,^[2] or hidden web^[3] are parts of the World Wide Web whose contents are not indexed by standard web search engines for any reason.^[4] The opposite term to the deep web is the surface web, which is accessible to anyone using the Internet.^[5] Computer scientist Michael K. Bergman is credited with coining the term deep web in 2001 as a search indexing term.^[6]

The content of the deep web is hidden behind HTTP forms,^[7][8] and includes many very common uses such as web mail, online banking, and services that users must pay for, and which is protected by a paywall, such as video on demand, some online magazines and newspapers, and many more.

Content of the deep web can be located and accessed by a direct URL or IP address, and may require password or other security access past the public website page.

Terminology[edit]

The first conflation of the terms "deep web" and "dark web" came about in 2009 when the deep web search terminology was discussed alongside illegal activities taking place on the Freenet darknet.^[9]

Since then, the use in the Silk Road's media reporting, many^[10][11] people and media outlets, have taken to using Deep Web synonymously with the dark web or darknet, a comparison many reject as inaccurate^[12] and consequently is an ongoing source of confusion.^[13] Wired reporters Kim Zetter^[14] and Andy Greenberg^[15] recommend the terms be used in distinct fashions. While the deep web is reference to any site that cannot be accessed through a traditional search engine, the dark web is a portion of the deep web that has been intentionally hidden and is inaccessible through standard browsers and methods.^{[16][17][18][19][20]}

Non-indexed content[edit]

Bergman, in a paper on the deep web published in The Journal of Electronic Publishing, mentioned that Jill Ellsworth used the term Invisible Web in 1994 to refer to websites that were not registered with any search engine.^[21] Bergman cited a January 1996 article by Frank Garcia:^[22]

It would be a site that's possibly reasonably designed, but they didn't bother to register it with any of the search engines. So, no one can find them! You're hidden. I call that the invisible Web.

Another early use of the term Invisible Web was by Bruce Mount and Matthew B. Koll of Personal Library Software, in a description of the #1 Deep Web tool found in a December 1996 press release.^[23]

The first use of the specific term deep web, now generally accepted, occurred in the aforementioned 2001 Bergman study.^[21]

Indexing methods[edit]

Methods which prevent web pages from being indexed by traditional search engines may be categorized as one or more of the following:

1. Contextual web: pages with content varying for different access contexts (e.g., ranges of client IP addresses or previous navigation sequence).

2. Dynamic content: dynamic pages which are returned in response to a submitted query or accessed only through a form, especially if open-domain input elements (such as text fields) are used; such fields are hard to navigate without domain knowledge.

3.Limited access content: sites that limit access to their pages in a technical way (e.g., using the Robots Exclusion Standard or CAPTCHAs, or no-store directive which prohibit search engines from browsing them and creating cached copies).^[24]

4.  Non-HTML/text content: textual content encoded in multimedia (image or video) files or specific file formats not handled by search engines.

5  Private web: sites that require registration and login (password-protected resources).

6.  Scripted content: pages that are only accessible through links produced by JavaScript as well as content dynamically downloaded from Web servers via Flash or Ajaxsolutions.

7. Software: certain content is intentionally hidden from the regular Internet, accessible only with special software, such as Tor, I2P, or other darknet software. For example, Tor allows users to access websites using the .onion server address anonymously, hiding their IP address.

8.  Unlinked content: pages which are not linked to by other pages, which may prevent web crawling programs from accessing the content. This content is referred to as pages without backlinks (also known as inlinks). Also, search engines do not always detect all backlinks from searched web pages.

9.Web archives: Web archival services such as the Wayback Machine enable users to see archived versions of web pages across time, including websites which have become inaccessible, and are not indexed by search engines such as Google.^[25]

Content types[edit]

Non Indexed Content

While it is not always possible to directly discover a specific web server's content so that it may be indexed, a site potentially can be accessed indirectly (due to computer vulnerabilities).

To discover content on the web, search engines use web crawlers that follow hyperlinks through known protocol virtual port numbers. This technique is ideal for discovering content on the surface web but is often ineffective at finding deep web content. For example, these crawlers do not attempt to find dynamic pages that are the result of database queries due to the indeterminate number of queries that are possible.^[6] It has been noted that this can be (partially) overcome by providing links to query results, but this could unintentionally inflate the popularity for a member of the deep web.

DeepPeep, Intute, Deep Web Technologies, Scirus, and Ahmia.fi are a few search engines that have accessed the deep web. Intute ran out of funding and is now a temporary static archive as of July 2011.^[26] Scirus retired near the end of January 2013.^[27]

Researchers have been exploring how the deep web can be crawled in an automatic fashion, including content that can be accessed only by special software such as Tor. In 2001, Sriram Raghavan and Hector Garcia-Molina (Stanford Computer Science Department, Stanford University)^[28][29] presented an architectural model for a hidden-Web crawler that used key terms provided by users or collected from the query interfaces to query a Web form and crawl the Deep Web content. Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho of UCLA created a hidden-Web crawler that automatically generated meaningful queries to issue against search forms.^[30] Several form query languages (e.g., DEQUEL^[31]) have been proposed that, besides issuing a query, also allow extraction of structured data from result pages. Another effort is DeepPeep, a project of the University of Utah sponsored by the National Science Foundation, which gathered hidden-web sources (web forms) in different domains based on novel focused crawler techniques.^[32][33]

Commercial search engines have begun exploring alternative methods to crawl the deep web. The Sitemap Protocol (first developed, and introduced by Google in 2005) and OAI-PMH are mechanisms that allow search engines and other interested parties to discover deep web resources on particular web servers. Both mechanisms allow web servers to advertise the URLs that are accessible on them, thereby allowing automatic discovery of resources that are not directly linked to the surface web. Google's deep web surfacing system computes submissions for each HTML form and adds the resulting HTML pages into the Google search engine index. The surfaced results account for a thousand queries per second to deep web content.^[34] In this system, the pre-computation of submissions is done using three algorithms:

1.  selecting input values for text search inputs that accept keywords,

2.  identifying inputs which accept only values of a specific type (e.g., date), and

3.  selecting a small number of input combinations that generate URLs suitable for inclusion into the Web search index.

In 2008, to facilitate users of Tor hidden services in their access and search of a hidden .onion suffix, Aaron Swartz designed Tor2web—a proxy application able to provide access by means of common web browsers.^[35] Using this application, deep web links appear as a random string of letters followed by the .onion TLD.

Dark web

From Wikipedia, the free encyclopedia

This article is about darknet websites. For the part of the Internet not accessible by traditional web search engines, see Deep web.

The dark web is the World Wide Web content that exists on darknets, overlay networks that use the Internet but require specific software, configurations or authorization to access.^[1][2] The dark web forms a small part of the deep web, the part of the Web not indexed by web search engines, although sometimes the term deep web is mistakenly used to refer specifically to the dark web.^{[3][4][5][6][7]}

The darknets which constitute the dark web include small, friend-to-friend peer-to-peer networks, as well as large, popular networks like Tor, Freenet, I2P and Riffle operated by public organizations and individuals. Users of the dark web refer to the regular web as Clearnet due to its unencrypted nature.^[8] The Tor dark web may be referred to as onionland,^[9] a reference to the network's top-level domain suffix .onion and the traffic anonymization technique of onion routing.

Terminology[edit]

The dark web has often been confused with the deep web, which refer to the parts of the web not indexed (searchable) by search engines. This confusion dates back to at least 2009.^[10] Since then, especially in reporting on Silk Road, the two terms have often been conflated,^[11][12][13] despite recommendations that they should be distinguished.^{[5][14][15][16]}

Definition[edit]

Main article: Darknet

Darknet websites are accessible only through networks such as Tor ("The Onion Routing" project) and I2P ("Invisible Internet Project").^[17] Tor browser and Tor-accessible sites are widely used among the darknet users and can be identified by the domain ".onion".^[18] While Tor focuses on providing anonymous access to the Internet, I2P specializes on allowing anonymous hosting of websites.^[19] Identities and locations of darknet users stay anonymous and cannot be tracked due to the layered encryption system. The darknet encryption technology routes users' data through a large number of intermediate servers, which protects the users' identity and guarantees anonymity. The transmitted information can be decrypted only by a subsequent node in the scheme, which leads to the exit node. The complicated system makes it almost impossible to reproduce the node path and decrypt the information layer by layer.^[20] Due to the high level of encryption, websites are not able to track geolocation and IP of their users, and users are not able to get this information about the host. Thus, communication between darknet users is highly encrypted allowing users to talk, blog, and share files confidentially.^[21]

The darknet is also used for illegal activity such as illegal trade, forums, and media exchange for pedophiles and terrorists.^[22] At the same time traditional websites have created alternative accessibility for the Tor browser in efforts to connect with their users. ProPublica, for example, launched a new version of its website available exclusively to Tor users.^[23]

Content[edit]

Web-based hidden services in January 2015[24]
Category	Percentage
Gambling	0.4
Guns	1.4
Chat	2.2
New (not yet indexed)	2.2
Abuse	2.2
Books	2.5
Directory	2.5
Blog	2.75
Porn	2.75
Hosting	3.5
Hacking	4.25
Search	4.25
Anonymity	4.5
Forum	4.75
Counterfeit	5.2
Whistleblower	5.2
Wiki	5.2
Mail	5.7
Bitcoin	6.2
Fraud	9
Market	9
Drugs	15.4

Web-based hidden services in February 2016[25][26]
Category	% of total	% of active
Violence	0.3	0.6
Arms	0.8	1.5
Illicit Social	1.2	2.4
Hacking	1.8	3.5
Illicit links	2.3	4.3
Illicit pornography	2.3	4.5
Extremism	2.7	5.1
Illicit Other	3.8	7.3
Illicit Finance	6.3	12
Illicit Drugs	8.1	15.5
Non-illicit+Unknown	22.6	43.2
Illicit total	29.7	56.8
Inactive	47.7
Active	52.3

A December 2014 study by Gareth Owen from the University of Portsmouth found that the most commonly hosted type of content on Tor was child pornography, followed by black markets, while the individual sites with the highest traffic were dedicated to botnet operations (see attached metric).^[27] Many whistleblowing sites maintain a presence^[28] as well as political discussion forums.^[29] Sites associated with Bitcoin, fraud related services and mail order services are some of the most prolific.^[27] To counter the trend of controversial content, the artist collective Cybertwee held a bake sale on an onion site.^[30]

In July 2017, Roger Dingledine, one of the three founders of the Tor Project, said that Facebook is the biggest hidden service. The Dark Web comprises only 3% of the traffic in the Tor network.^[31]

A more recent February 2016 study from researchers at King's College London gives the following breakdown of content by an alternative category set, highlighting the illicit use of .onion services.^[32][33]

Botnets[edit]

Botnets are often structured with their command and control servers based on a censorship-resistant hidden service, creating a large amount of bot-related traffic.^[27][34]

Bitcoin services[edit]

Bitcoin services such as tumblers are often available on Tor, and some – such as Grams – offer darknet market integration.^[35][36] A research study undertaken by Jean-Loup Richet, a research fellow at ESSEC, and carried out with the United Nations Office on Drugs and Crime, highlighted new trends in the use of Bitcoin tumblers for money launderingpurposes. A common approach was to use a digital currency exchanger service which converted Bitcoin into an online game currency (such as gold coins in World of Warcraft) that will later be converted back into money.^[37][38]

Darknet markets[edit]

Main article: Darknet market

Commercial darknet markets, which mediate transactions for illegal drugs^[39] and other goods, attracted significant media coverage starting with the popularity of Silk Road and Diabolus Market^[40] and its subsequent seizure by legal authorities.^[41]Other markets sell software exploits^[42] and weapons.^[43] Examination of price differences in Dark web markets versus prices in real life or over the World Wide Web have been attempted as well as studies in the quality of goods received over the Dark web. One such study was performed on Evolution, one of the most popular crypto-markets active from January 2013 to March 2015.^[44] Although it found the digital information, such as concealment methods and shipping country, "seems accurate", the study uncovered issues with the quality of illegal drugs sold in Evolution, stating that, "... the illicit drugs purity is found to be different from the information indicated on their respective listings."^[44] Less is known about consumer motivations for accessing these marketplaces and factors associated with their use.^[45]

Hacking groups and services[edit]

Many hackers sell their services either individually or as a part of groups.^[46] Such groups include xDedic, hackforum, Trojanforge, Mazafaka, dark0de and the TheRealDeal darknet market.^[47] Some have been known to track and extort apparent pedophiles.^[48] Cyber crimes and hacking services for financial institutions and banks have also been offered over the Dark web.^[49] Attempts to monitor this activity have been made through various government and private organizations, and an examination of the tools used can be found in the Procedia Computer Science journal.^[50] Use of Internet-scale DNS Distributed Reflection Denial of Service (DRDoS) attacks have also been made through leveraging the Dark Web.^[51] There are many scam .onion sites also present which end up giving tools for download that are infected with trojan horses or backdoors.

Fraud services[edit]

There are numerous carding forums, PayPal and Bitcoin trading websites as well as fraud and counterfeiting services.^[52] Many such sites are scams themselves.^[53]

Hoaxes and unverified content[edit]

Main article: Hoax

There are reports of crowdfunded assassinations and hitmen for hire,^[43][54] however, these are believed to be exclusively scams.^[55][56] The creator of Silk Road, Ross Ulbricht, was arrested by Homeland Security investigations (HSI) for his site and allegedly hiring a hitman to kill six people, although the charges were later dropped.^[57][58]

There is an urban legend that one can find live murder on the dark web. The term "Red Room" has been coined based on the Japanese animation and urban legend of the same name. However, the evidence points toward all reported instances being hoaxes.^[59][60]

On June 25, 2015, a creepy indie game Sad Satan was reviewed by Youtubers Obscure Horror Corner which they claimed to have found via the dark web. Various inconsistencies in the channel's reporting cast doubt on the reported version of events.^[61] There are several websites which analyze and monitor the deep web and dark web for threat intelligence, for example Sixgill.^[62]

Phishing and scams[edit]

Phishing via cloned websites and other scam sites are numerous,^[63][64] with darknet markets often advertised with fraudulent URLs.^[65][66]

Puzzles[edit]

Puzzles such as Cicada 3301 and successors will sometimes use hidden services in order to more anonymously provide clues, often increasing speculation as to the identity of their creators.^[67]

Illegal pornography[edit]

There is regular law enforcement action against sites distributing child pornography^[68][69] – often via compromising the site by distributing malware to the users.^[70][71] Sites use complex systems of guides, forums and community regulation.^[72] Other content includes sexualised torture and killing of animals^[73] and revenge porn.^[74]

Terrorism[edit]

There are at least some real and fraudulent websites claiming to be used by ISIL (ISIS), including a fake one seized in Operation Onymous.^[75] In the wake of the November 2015 Paris attacks an actual such site was hacked by an Anonymous affiliated hacker group GhostSec and replaced with an advert for Prozac.^[76] The Rawti Shax Islamist group was found to be operating on the dark web at one time.^[77]

Social media[edit]

Within the dark web, there exist emerging social media platforms similar to those on the World Wide Web. Facebook and other traditional social media platforms have begun to make dark-web versions of their websites to address problems associated with the traditional platforms and to continue their service in all areas of the World Wide Web.^[23]

Commentary[edit]

Although much of the dark web is innocuous, some prosecutors and government agencies, among others, are concerned that it is a haven for criminal activity.^[78] Specialist news sites such as DeepDotWeb^[79][80] and All Things Vice^[81] provide news coverage and practical information about dark web sites and services. The Hidden Wiki and its mirrors and forks hold some of the largest directories of content at any given time.

Popular sources of dark web .onion links include Pastebin, YouTube, Twitter, Reddit and other Internet forums.^[82] Specialist companies with Darksum and Recorded Future track dark web cybercrime goings on for law enforcement purposes.^[83] In 2015 it was announced that Interpol now offers a dedicated dark web training program featuring technical information on Tor, cybersecurity and simulated darknet market take downs.^[84]

In October 2013 the UK's National Crime Agency and GCHQ announced the formation of a 'Joint Operations Cell' to focus on cybercrime.^[85] In November 2015 this team would be tasked with tackling child exploitation on the dark web as well as other cybercrime.^[86]

In March 2017 the Congressional Research Service released an extensive report on the dark web, noting the changing dynamic of how information is accessed and presented on it; characterized by the unknown, it is of increasing interest to researchers, law enforcement, and policymakers.^[87]

In August 2017, according to reportage, cybersecurity firms which specialize in monitoring and researching the dark web on behalf of banks and retailers routinely share their findings with the FBI and with other law enforcement agencies "when possible and necessary" regarding illegal content. The Russian-speaking underground offering a crime-as-a-service model is regarded as being particularly robust.^[88]

Journalism[edit]

Many individual journalists, alternative news organizations, and educators or researchers are influential in their writing and speaking of the Darknet, and making its use clear to the general public.

Jamie Bartlett[edit]

Jamie Bartlett is a journalist and tech blogger for The Telegraph and Director of the Centre for the Analysis of Social Media for Demos in conjunction with The University of Sussex. In his book, The Dark Net,^[89] Barlett depicts the world of the Darknet and its implications for human behavior in different context. For example, the book opens with the story of a young girl who seeks positive feedback to build her self-esteem by appearing naked online. She is eventually traced on social media sites where her friends and family were inundated with naked pictures of her. This story highlights the variety of human interactions the Darknet allows for, but also reminds the reader how participation in a overlay network like the Darknet is rarely in complete separation from the larger Web. Bartlett's main objective is an exploration of the Darknet and its implication for society. He explores different sub-cultures, some with positive implications for society and some with negative.^[90]

Bartlett gave a TEDTalk in June 2015 further examining the subject.^[91] His talk, entitled "How the mysterious Darknet is going mainstream", introduces the idea behind the Darknet to the audience, followed by a walkthrough example of one of its websites called the Silk Road. He points out the familiarity of webpage design similar to consumer sites used in the larger commercial Web. Bartlett then presents examples of how operating in an uncertain, high-risk market like those in the Darknet actually breeds innovation that he believes can be applied to all markets in the future. As he points out, because vendors are always thinking of new ways to get around and protect themselves, the Darknet has become more decentralized, more customer friendly, harder to censor, and more innovative. As our societies are increasingly searching for more ways to retain privacy online, such changes as those occurring in the Darknet are not only innovative, but could be beneficial to commercial online websites and markets.

Other media[edit]

Traditional media and news channels like ABC News have also featured articles examining the Darknet.^[92] Vanity Fair magazine published an article in October 2016 entitled "The Other Internet". The article discusses the rise of the Dark Net and mentions that the stakes have become high in a lawless digital wilderness. It mentions that vulnerability is a weakness in a network's defenses. Other topics include the e-commerce versions of conventional black markets, cyberweaponry from TheRealDeal, and role of operations security.^[93]