Deep Web (Gilusis Internetas): kas tai ir ką verta žinoti

Exploring the Hidden Depths of the Internet. Deep web

The term "deep web" often brings to mind images of secret and illegal activities. However, the reality of the deep web is much more ordinary and extensive. Known also as the hidden web, it represents a vast part of the internet that standard search engines like Google, Bing, or Yahoo do not index. This section of the internet includes web pages and databases that can't be accessed by regular search methods, covering everything from academic journals and private databases to subscription-based services.

Understanding the Deep Web

The deep web includes everything from medical records and membership websites to paid content and private academic databases. To access content on the deep web, one typically needs direct links or specific credentials, making it a crucial resource for various professional and personal uses. Contrary to popular belief, most of the deep web is used for legitimate purposes.

History and Scale

The concept of the deep web has been around since the development of ARPANet in the 1970s, a project that laid the groundwork for today’s internet. It gained more public attention after the publication of "The Darknet and the Future of Content Distribution" in 2002 by Microsoft researchers. These researchers detailed parts of the internet inaccessible through regular means, focusing on content distribution networks.

Standard search engines like Google index only about 0.03% of the internet. The rest, which lies in the deep web, requires special tools and permissions to access due to its non-indexed nature.

The Technology Behind the Deep Web

A key player in accessing the "public" deep web is the Tor network. Originally developed by the U.S. military, Tor is free and open-source software that supports anonymous communication across a network of servers. Tor employs "onion routing," a method where communications are encrypted and relayed through multiple nodes, making the user's location and the information being accessed virtually untraceable.

1. How the Deep Web Stays Hidden

• Dynamic Content: Much of the deep web is made up of dynamic pages that are generated in response to queries. This means they exist only when needed and disappear afterwards, unlike static pages indexed by search engines.
• Robots.txt Files: Many websites use a file called "robots.txt" to prevent search engines from indexing their pages. This file is a directive to web crawlers, specifying what they can and cannot look at, effectively making parts of the site part of the deep web.
• No Hyperlinks: Some deep web pages have no links pointing to them from other sites, making them invisible to search engines that crawl the internet by following links between sites.

In the early 2000s, Google attempted to index the deep web. The endeavor, known as "Deep Crawling," wasn't entirely successful due to the vastness of the deep web and the dynamic nature of its content. This highlighted the complexity and ever-evolving landscape of the internet beneath the surface.

2. Encryption and Anonymity

• Encryption Layers: On networks like Tor, data is encrypted multiple times as it passes through a series of nodes or relays. Each node peels off one layer of encryption, akin to layers of an onion, which is why it’s called onion routing.
• Random Paths: The routing of data packets through random paths in the Tor network ensures that the origin, destination, and content of the communication are kept anonymous. This random path through nodes protects users from surveillance and traffic analysis.

3. Accessing the Deep Web

• Specialized Browsers: Accessing the deep web often requires specialized software like the Tor browser, which can access both the surface web and deep websites while maintaining user anonymity and security.
• Secure Directories: While not indexed by common search engines, some directories on the deep web provide listings of websites, similar to a phone book, but access to these directories often requires specific permissions or knowledge of their existence.

4. Uses of the Deep Web for Data Protection

• Personal Data: Many organizations store sensitive personal data (like medical records, and financial information) on the deep web to restrict access and protect privacy.
• Corporate Firewalls and VPNs: Corporations use the deep web internally to house information behind firewalls and VPNs, ensuring that sensitive corporate information (like internal communications, and proprietary data) is accessible only to authorized personnel.

5. Impact on Internet Bandwidth and Architecture

• Storage and Scale: The deep web, due to its massive size, affects the storage and architecture of the internet. Data centers and server configurations are often designed considering the vast amount of data stored in the deep web, which is rarely accessed but must be kept secure and ready for access when needed.

Legal Uses and Misconceptions

Aside from its use by individuals, to protect personal privacy, Tor and similar technologies are critical for journalists communicating with sources in repressive countries, legal teams collecting sensitive information, and businesses conducting confidential operations.

It's crucial to differentiate the deep web from the "dark web," which is a small portion of the deep web associated with illegal activities. While the dark web has garnered much media attention, it is only a tiny part of the broader deep web. It serves many legitimate functions and is essential for a wide range of online activities.

Tor, which stands for "The Onion Router," was originally developed to protect U.S. intelligence communications online. It was created by the U.S. Naval Research Laboratory in the mid-1990s and released to the public in 2002, with its development funded by the U.S. government as a tool for anonymizing intelligence communications.

The deep web is not just a place of mystery and lawlessness as often portrayed by sensational media stories. It is a critical component of the internet that enables secure communications, access to detailed information, and the storage of a vast amount of data beyond the reach of standard search engines. Understanding the true nature and purpose of the deep web is crucial for recognizing its importance in our digital age. By exploring the deep web, we gain insight into the vast potential and resources available beyond the visible internet, underscoring the depth and complexity of the online world.

Navigating the deep web can be a valuable experience for accessing a vast array of information not readily available on the surface web. However, due to the complex nature of the deep web, it's crucial to approach it with a heightened sense of caution and awareness. Here are several practical tips to help ensure your explorations are safe and secure:

1. Understand the Differences: It's important to differentiate between the deep web, the dark web, and the surface web. While the deep web includes all unindexed data like private databases, the dark web is a small part that's often associated with heightened privacy and sometimes illicit activities.
2. Use the Right Tools: If exploring the dark web, the Tor browser is essential as it helps anonymize your browsing, which is crucial for protecting your privacy. Ensure that you download it from its official website to avoid counterfeit versions.
3. Stay Anonymous: Avoid sharing any personal information. This includes using a non-identifiable email for any registrations and not sharing your real name or location.
4. Employ Strong Security Measures: Maintain strong antivirus software and a robust firewall. It’s also beneficial to use security-centric operating systems, such as Tails, which are designed to preserve privacy and anonymity.
5. Be Skeptical of Downloads: Exercise extreme caution with downloads. Files from the deep web can potentially be harmful or contain malware, so downloading should be kept to a minimum and always be scanned with security software.
6. Keep Everything Updated: Regular updates to your browser, security software, and operating system are critical. These updates help protect against the exploitation of vulnerabilities by malicious entities.
7. Educate Yourself About Scams: The deep web has its share of scams. Offers that appear too good to be true likely are, so approach them with scepticism.

By adhering to these guidelines, you can navigate the deep web more safely, making the most of the extensive information it has to offer while mitigating potential risks.